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Historically  decision  makers  have  had  to  manage  the 
risks  of  software-intensive  programs  and  projects  by 
applying  heuristic  methods  based  on  their  own  individ¬ 
ual  skills  and  experience.  As  software-intensive  sys¬ 
tems  increase  in  complexity  and  size,  however,  it  is  es¬ 
sential  for  decision-makers  to  use  more  disciplined  and 
systematic  methods  for  managing  software  risks. 

By  using  defined  methods  that  are  suited  to  the  state  of 
the  practice,  decision  makers  obtain  the  insights  nec¬ 
essary  to  take  actions  that  are  often  crucial  to  the  suc¬ 
cess  of  their  programs  and  projects. 

To  effectively  manage  software  risks,  it  is  necessary  to 
use  defined  methods  in  many  areas  of  risk  manage¬ 
ment,  including  identification  and  analysis.  It  is  impera¬ 
tive  that  the  methods  be  able  to  facilitate  communica¬ 
tion  among  all  parties  and  at  all  levels  of  the  organiza¬ 
tion. 

In  addition,  it  is  necessary  that  the  methods  promote 
the  efficient  management  of  software  risks  and  provide 
decision  makers  with  sufficient  information  to  focus  on 
the  priority  risks  and  mitigate  them  with  the  scarce  re¬ 
sources  that  are  available  to  them. 

The  Software  Engineering  Institute  (SEI)  has  designed 
and  developed  the  Software  Risk  Evaluation  (SRE) 
method  as  one  of  the  methods  of  software  risk  man¬ 
agement  that  can  meet  the  specific  needs  of  decision¬ 
makers  who  are  responsible  for  managing  software-in¬ 
tensive  programs  or  projects. 


. . . 
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One  of  the  goals  of  the  developers  is  to  make  the  meth¬ 
od  flexible  and  adaptable  to  the  practical  needs  of  soft¬ 
ware-intensive  programs  and  projects.  A  simultaneous 
goal  is  to  ensure  the  method  is  also  well  defined  and 
can  be  applied  systematically  and  in  a  disciplined  man¬ 
ner  within  any  organization. 

The  SRE  method  is  based  on  the  foundational  work 
that  was  laid  within  the  Risk  Program  of  the  Software 
Engineering  Institute.  It  is  also  one  of  the  products  that 
facilitate  the  implementation  of  the  SEI  risk  manage¬ 
ment  paradigm. 

The  robustness  of  the  latest  version  of  the  SRE  method 
is  a  result  of  continuous  application  and  field  testing  on 
actual  software  programs  and  projects.  This  has  over 
time  both  improved  and  enhanced  the  method  to  the 
current  state  of  the  practice. 

The  functional  components  of  the  SRE  method  and  its 
implementation  phases  and  activities  are  designed  to 
be  effective  for  managing  the  software  risks  of  pro¬ 
grams  and  projects.  In  addition  the  integrated  tools  and 
techniques  of  the  SRE  method  enable  its  utility  as  an 
efficient  and  practical  method. 
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Software  Risk  Evaluation  Method 


Abstract 

The  Software  Risk  Evaluation  (SRE)  method  is  used 
for  identifying,  analyzing,  communicating,  and  mitigat¬ 
ing  software  risks. 

The  SRE  method  is  intended  to  be  used  by  decision¬ 
makers  for  managing  the  software  risks  of  software-in¬ 
tensive  programs  and  projects.  The  SRE  method  facil¬ 
itates  the  mitigation  of  software  risks  for  managers. 

This  document  reports  on  the  Software  Risk  Evaluation 
method  version  1 .0  and  provides  a  high-level  descrip¬ 
tion  of  the  method.  The  report  is  intended  to  provide  the 
reader  with  an  overview  of  the  functional  components 
and  the  implementation  aspects  of  the  method. 

The  current  version  of  the  SRE  method  evolved  from 
two  earlier  versions.  The  first  version  was  developed 
under  a  technical  collaboration  agreement  with  the  Di¬ 
rector  of  Defense  Research  &  Engineering  (DDR&E) 
for  mitigating  the  risks  inherent  in  software  acquisi¬ 
tions.  The  latter  version  was  developed  to  improve  and 
enhance  the  capabilities  of  the  method  for  users  in  soft¬ 
ware-intensive  programs  and  projects. 
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1  Introduction 


Purpose 


Scope 


1.1  Purpose  and  Scope 

The  purpose  of  this  document  is  to  provide  a  technical 
report  on  the  Software  Risk  Evaluation  (SRE)  method. 
The  report  describes  both  the  functional  components 
and  the  implementation  aspects  of  the  SRE  method. 

This  report  provides  an  overview  of  the  SRE  process¬ 
es,  activities,  tools,  and  techniques.  It  also  describes 
some  of  the  previous  work  done  at  the  Software  Engi¬ 
neering  Institute  (SEI)  that  was  foundational  for  the  de¬ 
velopment  of  the  SRE  method. 

The  SRE  technical  report  is  intended  to  provide  the 
reader  with  a  general  understanding  of  the  SRE  meth¬ 
od.  It  does  not  define  the  details  that  are  required  for 
the  practice  of  the  method.  The  implementation  details 
are  available  in  other  materials  such  as  SRE  hand¬ 
books,  guidebooks,  and  templates.  The  practical  as¬ 
pects  of  the  SRE  method  will  be  facilitated  through  a 
training  course  offered  by  the  SEI. 


The  scope  of  this  document  is  to  answer  the  following: 

•  What  is  the  SRE  method,  and  what  are  its 
functional  components? 

•  What  is  the  overall  process  for  applying  the 
SRE  method  in  practice? 
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1.2  Background 


DDR&E  Needs  In  1991  the  Director  of  Defense  Research  &  Engineer¬ 

ing  (DDR&E)  established  a  Software  Action  Plan  Work¬ 
ing  Group  (SWAP-WG)  to  identify  the  software-related 
areas  that  could  be  leveraged  by  the  timely  application 
of  resources. 

One  of  the  identified  areas  was  the  risks  associated 
with  software  acquisitions  and  the  need  to  identify  and 
mitigate  the  risks  at  an  early  stage.  The  SWAP-WG  de¬ 
cided  to  address  this  need  by  using  the  ongoing  work 
at  the  Software  Engineering  Institute  (SEI)  and  by  en¬ 
listing  the  expertise  that  was  available  within  the  SEI 
Risk  Program. 


Version  0.1  The  initial  document  (version  0.1 )  of  the  SRE  method 

was  developed  based  on  a  technical  agreement  that 
was  signed  between  the  SEI  and  the  DDR&E  in  1993. 

The  SRE  method  version  0.1  was  reviewed  and  suc¬ 
cessfully  field  tested  on  two  software  programs.  The 
field  tests  identified  areas  for  improvement  of  the  SRE 
method. 


Version  0.2  The  SRE  method  version  0.2  was  produced  in  early 

1994.  It  documented  the  improvements  that  were 
made  to  the  earlier  version.  The  documentation  includ¬ 
ed  modifications  that  improved  the  readability  as  well 
as  implementation  efficiency  of  the  method. 
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The  SRE  method  version  0.2  has  been  field  tested  on 
many  software-intensive  programs  and  projects.  With 
each  field  test,  one  aspect  or  another  of  the  method 
has  been  improved  or  enhanced. 

The  feedback  from  individuals  based  on  their  review  of 
version  0.2  documentation  also  enabled  many  im¬ 
provements,  particularly  to  the  structure  of  the  docu¬ 
ment. 


This  report  contains  the  enhancements  and  improve¬ 
ments  that  were  made  to  the  previous  SRE  version  0.2 
document. 

The  enhancements  to  the  previous  version  are  mainly 
in  the  SRE  commitment  process,  that  is,  the  steps  re¬ 
quired  prior  to  executing  the  method,  in  the  risk  mitiga¬ 
tion  functions,  and  in  associated  implementation. 

The  improvements  to  the  previous  version  are  a  result 
of  the  field  testing  and  validation  that  were  done  on  the 
SRE  method  application  on  a  variety  of  software-inten¬ 
sive  programs  and  projects.  This  report  also  includes 
improved  terminology  and  easy-to-understand  vocabu¬ 
lary  as  suggested  by  the  reviewers  of  the  previous  doc¬ 
ument  (version  0.2). 
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1.3  Strategic  Direction 


Long  Term  While  the  immediate  focus  of  the  development  team  is 

Goals  to  develop  and  continuously  improve  the  SRE  method, 

the  long-term  goals  are  to  ensure  that 

•  The  method  is  defined  and  can  be  applied  in  a 
systematic,  disciplined,  and  efficient  manner. 

•  For  a  specific  program  or  project,  at  any  given 
instance  of  its  application,  there  is  uniformity  in 
the  outcome  of  the  risk  findings  and  mitigation. 

•  It  is  flexible  enough  to  be  used  in  different  sit¬ 
uations  and  phases  of  the  life  cycle,  including 
acquisition  and  maintenance. 


Duality  in  To  meet  the  long-term  goals,  the  SEI  has 

Approach  adopted  a  dual  and  evolutionary  path  of  development 

for  the  SRE  method.  The  first  path  concerns  the  devel¬ 
opment  of  the  method  itself  in  terms  of  its  processes, 
techniques,  and  tools,  and  is  documented  in  this  report. 

The  second  path  concerns  the  capture  of  knowledge 
from  various  domain  experts  to  build  a  predictive  deci¬ 
sion  model  for  proactive  management  of  software 
risks. 


Evolutionary  The  evolutionary  development  cycle  of 

Development  improvements,  field  testing,  and  validation  enables  the 

SRE  method  to  become  comprehensive  and  defined. 

The  evolutionary  development  provides  the  required 
maturity  in  the  operational  details  and  ensures  existing 
techniques  and  tools  are  enhanced  and  new  ones  are 
added  to  meet  the  practitioner’s  needs. 
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Model 


Knowledge  Domains 


The  development  of  a  predictive  decision  model  for 
software  risks  is  of  strategic  interest  to  the  SEI  and  the 
user  community.  This  challenging  task  requires  the 
employment  and  use  of  knowledge  engineering  tech¬ 
niques. 

The  premise  of  the  predictive  decision  model  as  stated 
in  Morgan  [Morgan  81]  and  others  is  that  although  good 
risk  data  may  not  be  available  for  a  particular  program 
or  project,  the  data  available  from  other  similar  pro¬ 
gram  or  project  experiences  may  be  adapted  or  ex¬ 
tended  for  use  either  directly  or  as  part  of  a  general 
model  for  predicting  risks  and  their  associated  mitiga¬ 
tion  efforts. 


Knowledge  domains  of  the  predictive  decision  model 
pertain  to  areas  such  as  the  characteristics  of  a  soft- 
ware-intensive  program  or  project,  typical  structures  of 
software  risks  and  their  drivers,  interactions  among 
software  risks  and  their  cause-and-effect  relationships. 

Knowledge  also  pertains  to  domains  that  are  required 
for  the  understanding  and  capture  of  the  cognitive  and 
intellectual  means  that  people  use  to  process  and  com¬ 
municate  risks. 

The  development  of  a  predictive  decision  model  for  risk 
management  requires  the  use  of  knowledge  engineer¬ 
ing  principles  and  techniques  such  as  protocol  analysis 
[Poltrock  89]. 


CMU/SEI-94-TR-1 9 


7 


8 


CMU/SEI-94-TR-1 9 


2  Technical  Basis 


Paradigm 


2.1  Risk  Management 


The  SEI  risk  management  paradigm  defines  a  continu¬ 
ous  set  of  activities  that  must  be  undertaken  to  identify, 
communicate,  and  resolve  software  risks. 

The  model  used  to  represent  this  paradigm  is  a  circular 
set  of  activities  emphasizing  that  risk  management  is  a 
continuous  process  (see  Figure  2-1 ). 

The  arrows  within  the  model  represent  the  logical  path 
and  the  sequence  in  flow  of  information  within  the  risk 
management  paradigm. 


Figure  2-1 :  Risk  Management  Paradigm 
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Activities 


The  activities  in  the  software  risk  management  para¬ 
digm  are: 

Identify:  Surfacing  software-related  risks  before  they 
become  problems  that  can  adversely  affect  the  pro¬ 
gram  or  project. 

Analyze:  Transforming  data  from  the  identified  risks 
into  decision  making  information. 

Plan:  Using  the  risk  information  in  decisions  and  ac¬ 
tions  including  developing  actions  to  mitigate  individual 
risks,  prioritizing  actions,  and  integrating  them  into  an 
executable  risk  management  plan. 

Track:  Monitoring  the  status  of  risks  and  their  mitiga¬ 
tion  actions  along  with  the  use  of  metrics  and  triggering 
events. 

Control:  Correcting  the  deviations  from  planned  risk 
mitigation  actions  by  using  existing  program  or  project 
management  control  functions. 

Communicate:  Exchanging  risk  management  informa¬ 
tion  among  the  functions  and  at  all  levels  of  the  organi¬ 
zation.  This  activity  is  represented  in  the  center  of  the 
model  to  emphasize  its  pervasiveness  and  criticality  for 
implementing  the  other  activities  in  the  paradigm. 
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Risk  Taxonomy 


Classes 


Elements 


2.2  Software  Risk  Taxonomy 

The  SEI  software  risk  taxonomy  provides  a  basis  for  or¬ 
ganizing  and  studying  various  aspects  of  software  risks 
in  a  program  or  project.  It  serves  not  only  as  a  system¬ 
atic  way  of  eliciting  and  organizing  risks  but  also  as  a 
consistent  framework  for  the  development  of  risk  man¬ 
agement  methods  and  techniques.  The  taxonomy  has 
been  developed  over  the  past  three  years  and  has 
been  validated  in  over  30  instances. 


The  taxonomy  is  organized  into  three  major  classes  as 
follows: 

Product  Engineering:  covers  the  technical  aspects  of 
the  work  to  be  accomplished. 

Development  Environment:  covers  the  methods,  pro¬ 
cedures,  and  tools  used  in  the  production  of  the  soft¬ 
ware  product. 

Program  Constraints:  covers  the  contractual,  organi¬ 
zational,  and  operational  factors  within  which  the  soft¬ 
ware  must  be  developed  and  that  are  generally  not  un¬ 
der  the  direct  control  of  the  organization. 


The  classes  in  the  taxonomy  are  divided  into  elements ; 
for  example,  the  Product  Engineering  class  is  divided 
into  Requirements,  Design,  Code  &  Unit  Test,  Integra¬ 
tion  and  Test,  and  Engineering  Specialties  elements. 
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Attributes  Each  element  is  further  divided  into  attributes-,  for  ex¬ 

ample,  the  Requirements  element  is  divided  into  the  at¬ 
tributes  of  Stability,  Completeness,  Clarity,  Validity,  Fa¬ 
miliarity,  Precedent,  and  Scale. 


Taxonomy  Structure  Figure  2-2  shows  the  structure  of  the  software  risk  tax¬ 
onomy. 


Software  Risks 


■m&v: 


Development 

Environment 


Product 

Engineering 


Program 

Constraints 


Stability  *Scale 


Schedule*  ‘Facilities 


Formality,  grpducjl 


Figure  2-2:  Structure  of  the  Software  Risk  Taxonomy 
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Purpose 


Typical  Usage 


2.3  Taxonomy-Based  Questionnaire 

The  Taxonomy-Based  Questionnaire  (TBQ)  is  a  tool 
used  to  identify  software  risks  in  a  program  or  project. 

The  purpose  of  this  tool  is  to  ensure  coverage  of  all  po¬ 
tential  risk  areas  by  asking  questions  at  a  detailed  at¬ 
tribute  level  of  the  software  risk  taxonomy. 


The  TBQ  contains  specific  questions  to  find  risks,  and 
follow-up  questions  and  cues  that  enable  the  person 
administering  the  questionnaire  to  probe  for  risks. 

The  TBQ  is  effective  when  used  along  with  appropriate 
interview  techniques  and  when  applied  on  manage¬ 
ment  and  technical  groups  within  an  organization.  The 
TBQ  is  also  found  to  be  more  effective  when  adminis¬ 
tered  by  an  independent  team  on  peer  groups. 

Since  the  TBQ  is  a  general  tool,  some  of  its  questions 
may  not  be  applicable  to  specific  programs  or  projects. 
The  TBQ  is  therefore  tailored  to  suit  the  specific  needs 
of  an  organization  before  it  is  administered. 
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TBQ  Conventions 


The  TBQ  uses  specific  conventions  to  facilitate  effec¬ 
tive  and  consistent  administering  of  the  questions.  For 
example,  a  question  that  is  framed  within  square  brack¬ 
ets  (see  figure  2-3)  is  used  as  a  “starter"  to  provide  a 
generalized  context  for  the  respondents.  Depending  on 
the  initial  response,  each  question  may  have  additional 
probe  questions. 


A.  Product  Engineering 
1.  Requirements 

a.  Stability 

[Are  requirements  changing  even  as  the  product  is  beina 
produced?]  y 

[1]  Are  the  requirements  stable? 

(No)  (1  .a)  What  is  the  effect  on  the  system? 

•  Qualify 

.  Functionality 

.  Schedule 
.  Integration 

•  Design 

•  Testing 

[2]  Are  the  external  interfaces  changing? 


Figure  2-3:  Taxonomy-Based  Questionnaire  View 


The  probe  questions  are  prefaced  in  parentheses  with 
“Yes”  or  “No,"  indicating  the  type  of  initial  response  that 
should  activate  its  use  during  the  interview. 

Some  questions  may  also  have  “cues”  (or  a  list  of  items 
in  bullet  format).  These  cues  are  used  to  trigger 
thoughts  in  the  minds  of  the  respondents  on  issues, 
concerns,  or  risks. 
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Introduction 


Sponsor 


System  View 


Risk  Areas 


iiiasaaiagimagm''^^ 

The  SRE  method  is  designed  to  be  a  flexible  method 
that  can  be  suitably  applied  in  a  variety  of  situations 
both  in  a  development  environment  and  in  an  acquisi¬ 
tion  environment.  The  SRE  method  can  also  be  suit¬ 
ably  applied  during  any  phase  of  the  development  or 
acquisition  life  cycle. 


Successful  application  of  the  method  depends  on  the 
availability  of  sponsorship.  The  sponsor  ensures  that 
the  goals  for  applying  the  SRE  method  are  attained. 
Typically  the  sponsor  of  a  software  risk  evaluation  is 
one  of  the  following: 

•  program  or  project  manager 

•  acquisition  manager 

•  corporate  manager 

•  individual  change  agent 


3.1  System  Environment 


The  SRE  method  views  risks  from  a  system  level.  It 
deals  with  the  many  uncertainties  associated  with  the 
development  and  acquisition  of  complex  software-in¬ 
tensive  programs  and  projects. 


Representative  areas  of  risk  in  a  system  environment 
are  software,  hardware,  technology,  cost  and  sched¬ 
ule,  and  people.  The  risks  in  these  areas  have  complex 
interactions  and  interdependencies  with  each  other. 
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Application 


Figure  3-1  shows  the  risk  areas  and  their  inter¬ 
relationship  in  a  system  environment. 


Figure  3-1 :  Risks  Within  a  System  Environment 


3.2  Sample  Application 


The  SRE  can  be  applied  in  different  situations  and 
within  different  environments.  One  example  of  an  SRE 
application  is  shown  in  Figure  3-2.  Here  the  SRE 
method  is  applied  in  a  program  where  a  government 
buyer-contractor  supplier  relationship  exists. 

In  this  scenario,  the  program  manager  or  client  recruits 
an  independent  team  to  perform  an  SRE.  The 
independent  team  executes  the  SRE  functions  and 
provides  a  data  confirmation  briefing  and  feedback  to 
the  contractor  organization. 


filial 
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Executing  the  SRE  produces  data  pertaining  to  risks 
the  independent  team  analyzes  within  the  context  of 
program  or  project  goals  and  environment.  By  working 
together  with  the  representatives  from  the  respective 
organizations,  the  SRE  team  develops  mitigation  strat¬ 
egies  and  specific  activities  for  managing  the  risks. 

The  results  of  the  risk  evaluation  are  then  provided  to 
the  client  in  a  final  report.  The  final  report  contains  both 
the  risk  findings  and  suggestions  to  mitigate  them. 

Note  that  the  application  of  the  SRE  method  is  not  lim¬ 
ited  to  this  scenario.  The  SRE  method  can  be  applied 
in  a  variety  of  situations,  for  example,  the  software  de¬ 
veloper  or  contractor  can  apply  the  method  internally 
as  a  tool  for  managing  software  risks. 


Figure  3-2:  Sample  Application 


Applications 


Timeline 


3.3  Notional  Timeline 


The  notional  timeline  for  conducting  a  software  risk 
evaluation  is  shown  in  Figure  3-3. 


Weeks 

Weeks 

Weeks 

Slllllllllll 

• 

A  A 

A 
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1  " 
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Team 

Training 

\/ 

On-Site 
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Report 

/  Final 

/  Report 

\ 

Establish  Goals 
Tailor  Method, 

&  Finalize  Plan 

/ 

Mitigation 

Activities 

Figure  3-3:  Notional  Timeline 


For  the  sample  application  that  was  described  in  the 
previous  section,  the  approximate  schedule  is  as 
follows: 


•Client  Agreement 

2  or  3  weeks 

•Establish  Goals,  Tailor 

Method,  and  Finalize  Plan 

1  or  2  weeks 

•Team  Training 

2  days 

•On-Site  Period 

1  week 

•  Preliminary  Report 

3  or  4  weeks 

•Mitigation  Activities 

1  week 

•Deliver  Final  Report 

1  or  2  weeks 
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4  Functional  Components 


4.1  Overview 


The  Software  Risk  Evaluation  method  consists  of  two 
types  of  functions  -  primary  and  support  functions  (see 
Figure  4-1). 

The  primary  functions  are  Detection,  Specification,  As¬ 
sessment,  Consolidation,  and  Mitigation.  The  support 
functions  are  Commitment,  Planning  &  Coordination, 
Verification  &  Validation,  and  Training  &  Communica¬ 
tion. 
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Figure  4-1 :  Functional  Components 
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Purpose 


Key  Items 
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4.2  Primary  Functions 


4.2.1  Detection 


Detection  is  the  function  of  finding  the  software  risks 
associated  with  a  software-intensive  program  or 
project. 

This  function  is  performed  to  ensure  systematic  and 
broad  coverage  of  all  potential  risk  areas.  It  is  also  per¬ 
formed  to  ensure  efficiency  and  effectiveness  in  finding 
software  risks  through  the  use  of  appropriate  tools  and 
techniques. 


Risk  detection  in  the  SRE  method  is  performed  by  us¬ 
ing  the  following  key  items: 

•  SEI  Taxonomy-Based  Questionnaire  to  en¬ 
sure  complete  coverage  of  all  areas  of  poten¬ 
tial  software  risks. 

•  An  independent  team  conducting  peer  group 
interviews  using  the  specified  process  and 
techniques. 

•  Selection  of  the  peer  groups  and  the  appropri¬ 
ate  individuals  within  each  group  using  the 
SRE  guidelines. 

•Training  of  the  independent  SRE  team  in  the 
SRE  method  and  the  effective  use  of  its  tools 
and  techniques. 
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Purpose 


Representations 


Representation-1 


4.2.2  Specification 


Risk  specification  is  the  function  of  recording  all  as¬ 
pects  of  the  identified  software  risks  including  its  condi¬ 
tions,  consequences,  and  immediate  source  of  the  risk 
for  taking  corrective  actions. 


There  are  many  ways  of  representing  software  risks 
that  range  from  simple  statement  of  the  issue  to  a  com¬ 
prehensive  specification  statement. 

Two  representations  are  provided  as  examples  below. 
Both  have  been  field  tested  and  found  to  be  effective  in 
practice. 


Figure  4-2  shows  an  example  of  a  simple  statement  of 
issue  with  the  software  risk  implied. 


Changes  to  the  logical  database  design  during 
coding  phaseare  causing  changes  to  program 
modules  that  access  the  database. 


Figure  4-2:  Simple  Representation 
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Representation-2 


Gluch  [Gluch  93]  has  developed  a  structured  represen¬ 
tation  for  software  risk  statements.  The  syntactical  con¬ 
struction  and  an  illustration  of  its  use  are  shown  in  Fig¬ 
ure  4-3. 

The  purpose  of  the  structured  representation  is  to: 

•  Serve  as  the  guiding  structure  to  identify  and 
communicate  risks. 

•  Capture  the  components  of  the  risk  and  to 
some  degree  simplify  the  task  of  prioritizing. 

•  Isolate  and  record  the  condition  when  the  risk 
becomes  valid. 

•  Assist  in  the  identification  of  source(s)  of  the 
risk. 


Example: 

Given  that  the  GUI  must  be  coded  using 
X  Windows  and  there  is  no  expertise  within 
the  project  in  X  programming  then  (possibly) 
the  GUI  code  will  not  be  completed  on  time 
and  the  implementation  can  be  inefficient. 


Figure  4-3:  Structured  Representation 
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Source  of  Risk  Risk  specification  also  includes  identifying  the  source 

of  the  risk. 

In  the  SRE  method  each  risk  is  assigned  to  a  specific 
category  within  the  SEI  taxonomy  of  software  risks. 

The  source  of  each  risk  is  specified  by  tagging  it  to  a 
class,  element,  and  attribute  of  the  taxonomy.  For  ex¬ 
ample,  if  the  immediate  source  of  risk  is  determined  to 
be  unstable  requirements  it  is  tagged  as  “A1  a”  (see 
Figure  A-1  for  taxonomy  groups). 


4.2.3  Assessment 


Purpose  Risk  assessment  is  a  function  that  is  performed  to  de¬ 

termine  the  magnitude  of  each  software  risk.  The  pri¬ 
mary  purpose  of  the  function  in  the  SRE  method  is  to 
prioritize  the  risks  and  to  effectively  mitigate  them. 


Definition  By  definition,  the  magnitude  of  a  risk  is  a  function  of  its 

severity  of  impact  and  the  probability  of  its  occurrence 
(see  Figure  4-4). 


m$k  Magnitude  *  severity  of  impact  *  Probability  of  occurrence 


Figure  4-4:  Definition  of  Risk  Magnitude 
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Mechanisms  There  are  different  mechanisms  that  can  be  used  for 

assessing  risk  magnitude  ranging  from  simple  subjec¬ 
tive  measures  to  more  rigorous  statistical  measures. 

In  the  current  version  of  the  SRE  method,  two  mecha¬ 
nisms  are  provided  as  examples  (see  below).  Both  are 
found  to  be  practical  and  suitable  for  performing  the 
risk  assessment  function. 


Mechanism-1  The  first  assessment  mechanism  uses  a  simple  scale 

where  risk  statements  are  assessed  at  one  of  four  lev¬ 
els  of  magnitude  depending  on  their  impact  on  the  fol¬ 
lowing  factors: 

•  Cost  and  schedule 

•  Performance 

•  Supportability 

Table  4-1  lists  the  four  levels  of  magnitude  and  their 
suggested  guidelines. 


Table  4.1 :  Levels  of  Magnitude  and  Guidelines 


Magnitude  s 

Guidelines 

Critical 

•  High  likelihood  of  the  risk  severely  impacting  one  or 
more  factors  i.e.,  cost  &  schedule,  performance,  and 
supportability 

High 

•  High  likelihood  of  the  risk  moderately  impacting  one  or 
more  factors 

Medium 

•  Medium  likelihood  of  the  risk  moderately  impacting  one 
or  more  factors 

Low 

•  Low  likelihood  of  the  risk  moderately  impacting  one  of 
more  factors 
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The  rationale  for  using  this  mechanism  is  as  follows: 

•  To  keep  the  scoring  mechanism  as  simple  as 
possible. 

•  To  use  a  subjective  scoring  mechanism  that 
can  be  effective  for  prioritizing  the  risks. 

•  To  surface  the  differences  in  opinion  among 
individual  team  members  regarding  the  mag¬ 
nitude  of  a  risk  and  to  reach  consensus. 

•  To  form  a  basis  for  conducting  the  technical 
consensus  discussions  during  the  analysis 
sessions. 


Mechanism-2  A  more  involved  mechanism  for  risk  assessment  is  one 

which  is  adapted  from  the  Air  Force  [Airforce  88]  and 
modified  for  use  in  the  SRE  method. 

In  this  mechanism,  risk  statements  are  assessed  at 
one  of  three  levels  of  magnitude: 

•  High 

•  Medium 

•  Low 

The  level  of  magnitude  of  a  particular  risk  depends  on 
the  assessment  of  its  severity  of  impact  and  its  proba¬ 
bility  of  occurrence. 


Severity  of  impact  is  the  effect  of  the  particular  risk  on 
the  target  program  or  project  and  is  determined  on  the 
basis  of  its  impact  on  the  software’s  performance,  sup- 
portability,  cost,  and  schedule. 
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Each  risk  is  determined  to  be  at  one  of  the  following 
levels  of  severity: 

•  Catastrophic 

•  Critical 

•  Marginal 


Probability  of  occurrence  is  the  certainty  or  likelihood 
of  the  risk  becoming  true  for  the  program  or  project. 

Each  risk  is  determined  to  be  at  one  of  the  following 
levels  of  probability: 

•  Very  Likely 

•  Probable 

•  Improbable 


The  matrix  shown  in  Figure  4-5  is  used  to  determine 
the  level  of  magnitude  based  on  assessment  of  the  se¬ 
verity  of  impact  and  the  probability  of  occurrence  of  a 
risk.  The  shaded  areas  indicate  different  levels  of  mag¬ 
nitude. 

For  example,  a  high  magnitude  risk  is  one  whose  se¬ 
verity  is  catastrophic  and  probability  of  occurrence  is 
very  likely  or  probable,  or  whose  severity  is  critical  and 
probability  of  occurrence  is  very  likely. 
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Figure  4-5:  Risk  Magnitude  Matrix 


4.2.4  Consolidation 


Purpose  This  function  is  performed  to  integrate  the  multiple  risk 

detection  activities  of  the  SRE  method  both  when  sim¬ 
ilar  risks  are  identified  from  different  sessions  and 
when  multiple  risk  evaluations  are  held  for  a  program 
or  project. 

For  example,  related  risks  may  be  identified  from  differ¬ 
ent  interview  sessions  during  a  site  visit.  Similarly,  re¬ 
lated  risks  may  be  identified  from  multiple  risk  evalua¬ 
tions  that  are  performed  for  a  program  or  project  or 
when  separate  site  visits  are  done  for  executing  the 
SRE  for  the  program  office  and  the  subcontractors 
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Criteria 


Guidelines 


who  are  responsible  for  developing  the  system  or  its 
components. 


The  risk  statements  that  are  considered  for  consolida¬ 
tion  must  meet  one  of  the  following  criteria: 

•  Manifestation  of  the  same  risk  statement,  that 
is,  identical  in  every  way  except  in  the  wording 
of  the  risk  statements. 

•  Fragmentation  due  to  minor  variations  or  dif¬ 
ferent  aspects  of  the  same  risk  statement. 

•  Differences  in  granularity,  for  example,  a  mi¬ 
nor  risk  statement  that  is  covered  in  the  con¬ 
text  of  another  risk  statement  of  larger  magni¬ 
tude. 


Some  of  the  suggested  guidelines  for  risk  consolidation 

activities  are  listed  below: 

•  Classify  related  risks  in  some  way,  such  as  by 
the  sources  of  risk  to  be  more  easily  identifi¬ 
able. 

•  Ensure  the  context  of  the  risk  is  maintained  by 
keeping  the  original  wording  of  the  affected 
risk  statements. 

•  Merge  identical  risk  statements  into  a  single 
statement. 

•Combine  fragmented  risk  statements  into  a 
single  statement  whenever  possible. 

•  Abstract  granular  risks  into  a  single  statement 
that  provides  sufficient  level  of  detail  for  effec¬ 
tive  risk  mitigation. 
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Purpose 


Mitigation  Area 


4.2.5  Mitigation 


Risk  mitigation  is  the  function  of  developing  strategies 
and  specific  activities  to  alleviate  and  eliminate  the 
threat  posed  by  risks  to  the  success  of  the  program  or 
project. 

The  mitigation  function  is  performed  by  identifying  ar¬ 
eas  where  the  program  or  project  can  focus  its  resourc¬ 
es  for  effectively  addressing  the  risks.  These  areas  are 
termed  mitigation  areas. 

Risk  mitigation  develops  the  strategies  and  activities 
with  reference  to  the  goals  of  the  program  or  project  for 
performing  the  risk  evaluation. 


A  risk  mitigation  area  is  a  logical  grouping  of  similar 
risks  to  enable  them  to  be  managed  efficiently  and  ef¬ 
fectively. 

The  use  of  risk  mitigation  areas  is  a  management  tech¬ 
nique  that  allows  development  and  implementation  of 
mitigation  activities  within  a  broader  perspective  of  pro¬ 
gram  or  project  goals  and  objectives. 

The  use  of  risk  mitigation  areas  has  been  found  to  be 
practical  particularly  when  dealing  with  a  large  number 
of  risks. 
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Figure  4-6  is  an  example  of  the  mitigation  areas  that 
were  identified  for  a  large-scale  integrated  database 
system. 


•  System  Requirements  Management 

•  Program  Planning  and  Tracking 

•  Program-Organization  Group  Coordination 

•  System  Configuration  Management 

•  System  Verification  and  Validation 


Figure  4-6:  Sample  Mitigation  Areas 


Risk  Map  A  risk  map  is  an  important  tool  that  is  used  for  the  de¬ 

velopment  of  mitigation  strategies  and  specific  mitiga¬ 
tion  activities  for  each  mitigation  area. 

The  risk  mapping  exercise  is  performed  by  mapping 
the  specific  risk  findings  that  were  identified  to  the  re¬ 
spective  mitigation  areas.  The  risk  map  should  also 
map  the  risk  findings  to  the  goals  of  the  program  or 
project  for  performing  the  risk  evaluation. 

The  risk  map  ensures  that  all  of  the  risks  are  being  ad¬ 
dressed  and  that  no  risk  is  omitted  inadvertently.  It  also 
ensures  that  risk  mitigation  will  address  client  goals. 
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Mitigation  Stages  The  mitigation  function  is  performed  in  four  stages  for 

each  mitigation  area  of  the  program  or  project. 

Table  4-2  lists  the  mitigation  stages  and  their  descrip¬ 
tions. 

Table  4.2:  Development  Stages  for  Mitigation  Activities 


Stage 

Description 

Analysis 

•  What  is  the  current  status?  A  description  of  the  current 
situation  of  the  program  or  project  within  each  mitiga¬ 
tion  area  is  developed.  The  risk  map  is  used  to  ensure 
all  risks  are  taken  into  consideration. 

Goals 

•  What  is  the  target  status?  Specific  goals  are  identified 
for  each  mitigation  area.  The  goals  collectively 
describe  the  desired  status  for  the  mitigation  area.  The 
goals  support  the  program  or  project  goals  and  objec¬ 
tives. 

Strategies 

•  What  can  be  done  to  reach  the  target  status?  A  strat¬ 
egy  is  developed  to  reach  the  desired  status  (goals)  for 
the  mitigation  area.  The  strategy  is  developed  as  a  set 
of  general  or  broad  recommendations  that  are  candi¬ 
dates  for  implementation  within  the  program  or  project. 
The  general  recommendations  that  are  to  be  imple¬ 
mented  within  the  program  or  project  are  selected  and 
prioritized  by  the  client  management. 

Activities 

•  What  activities  should  be  implemented?  Each  general 
recommendation  that  is  selected  for  implementation  is 
developed  into  specific  activities. 

Key  Activities:  Key  activities  to  implement  the  recom¬ 
mendation  are  first  identified.  If  there  are  any  con¬ 
straints  within  the  program  or  project,  the  activities  to 
overcome  the  constraints  are  also  identified. 

Enabling  Activities:  Enabling  activities  such  as  kick¬ 
off  meetings,  training,  developing  task  plans,  support 
needs,  etc.,  are  also  identified  for  implementation. 
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4.3  Support  Functions 


4.3.1  Commitment 


Purpose  The  commitment  function  is  very  important  for  the  suc¬ 

cessful  performance  of  the  software  risk  evaluation  and 
is  comprised  of  two  aspects. 

The  first  aspect  is  to  establish  a  business  relationship 
with  the  management  of  the  program  or  project  and  to 
understand  and  establish  the  goals  for  performing  the 
software  risk  evaluation.  The  second  aspect  is  to  gain 
the  commitment  in  the  constraint  of  how  the  SRE  appli¬ 
cation  will  satisfy  client  program  or  project  goals. 


Mechanism  Included  in  this  function  are  activities  that  can  provide 

executives  with  an  understanding  of  the  SRE  method 
in  terms  of  its  purpose,  scope,  required  resources,  ex¬ 
ecution  phase  activities,  mitigation  activities,  and  typi¬ 
cal  outcome  of  a  risk  evaluation. 

Also  included  are  activities  for  the  SRE  team  to  under¬ 
stand  the  program  or  project  goals  and  to  map  those  to 
the  activities  of  the  software  risk  evaluation. 


4.3.2  Planning  &  Coordination 


Purpose  Planning  and  coordination  is  a  supporting  function  of 

the  SRE  method.  Its  purpose  is  to  prepare  for  the  im¬ 
plementation  phases  of  the  evaluation. 
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Roles 


Purpose 


Mechanism 


Both  the  SRE  team  and  the  target  organization  are  in¬ 
volved  in  the  planning  and  coordination  function. 

SRE  team  leader  is  an  experienced  person  designat¬ 
ed  from  an  independent  organization  and  responsible 
for  the  overall  planning  and  coordination  of  the  SRE  im¬ 
plementation  activities. 

Site  coordinator  is  designated  from  the  target  organi¬ 
zation  to  be  the  single  interface  for  the  SRE  team  lead¬ 
er  to  perform  planning  and  coordination  activities  in¬ 
cluding  scheduling  individuals  for  interviews  and  ar¬ 
ranging  facilities  for  the  site  visit. 


4.3.3  Verification  &  Validation 


Verification  and  validation  is  a  supporting  function  of 
the  SRE  method. 

Its  purpose  is  to  ensure  quality  of  the  implementation 
process  and  the  accuracy  and  validity  of  the  results.  Its 
purpose  is  also  to  ensure  reliable  information  for  the  ef¬ 
fective  mitigation  of  software  risks. 


Included  in  this  function  are  activities,  tools,  and  tech¬ 
niques  that  facilitate  corrective  actions  from  the  early 
stages  of  implementation. 

For  example,  techniques  such  as  risk  play-back  and 
team  reviews  ensure  the  accuracy  of  the  content, 
structure,  attributes,  and  context  of  each  risk. 


Functional  Components 


Purpose 


Mechanism 


4.3.4  Training  &  Communication 


Training  and  communication  is  a  supporting  function  of 
the  SRE  method. 

Its  purpose  is  to  ensure  effectiveness  of  the  implemen¬ 
tation  process  by  ensuring  all  the  people  involved  have 
sufficient  knowledge,  understanding,  and  skills.  Its  pur¬ 
pose  is  also  to  create  an  environment  for  effective  dia¬ 
log  and  exchange  of  information  during  the  implemen¬ 
tation. 


Included  in  this  function  are  items  such  as  training  of 
SRE  team  members  who  will  be  involved  in  the  soft¬ 
ware  risk  evaluation,  orientation  of  management  and 
technical  personnel  associated  with  the  program  or 
project,  and  type  and  contents  of  briefings. 

Also  included  are  specific  activities,  tools,  and  tech¬ 
niques  that  are  used  during  the  implementation  pro¬ 
cess  to  ensure  proper  training  and  communication.  For 
example,  use  of  an  introduction  script  that  is  read  or 
paraphrased  by  the  interviewer  at  the  beginning  of 
each  interview  session  ensures  that  a  proper  environ¬ 
ment  is  set  for  the  open  communication  of  risks  where 
the  respondents  are  able  to  freely  discuss  any  issue  or 
concern. 
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Phases 


The  software  risk  evaluation  is  implemented  in  four 
phases  --  commitment,  preparation,  execution,  and 
mitigation. 

Commitment  consists  of  activities  to  establish  the 
need,  identify  program  or  project  goals,  and  obtain 
agreements  for  the  software  risk  evaluation. 

Preparation  consists  of  planning  and  coordination  ac¬ 
tivities  that  are  performed  prior  to  the  site  visit  for  exe¬ 
cuting  the  software  risk  evaluation. 

Execution  consists  of  activities  to  implement  the  SRE 
functions  during  a  site  visit  to  the  location  of  the  target 
program  or  project. 

Mitigation  consists  of  activities  to  mitigate  the  risks 
that  were  identified  during  the  software  risk  evaluation. 


Figure  5-1  shows  an  overview  of  the  implementation 
phases. 


Figure  5-1 :  Implementation  Phases  Overview 
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Introduction 


Introduction 


5.1  Commitment 


The  commitment  phase  requires  at  least  two  meetings 
and  the  completion  of  agreements  with  the  client  for 
performing  the  risk  evaluation. 

Figure  5-2  shows  the  sequence  of  activities  in  this 
phase. 


Figure  5-2:  Commitment  Phase  Activities 


5.1.1  First  Client  Meeting 


The  Software  Risk  Evaluation  process  begins  with  the 
client  recognizing  the  need  to  determine  the  risks  of  a 
software-intensive  program  or  project.  For  example,  a 
manager  may  want  to  determine  the  risks  of  the  soft¬ 
ware  development  effort  to  make  a  critical  decision  or 
as  a  routine  activity  in  the  practice  of  risk  management. 
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Participants  Usually  at  the  first  meeting  the  client’s  senior  manage¬ 

ment  and  designees  participate  in  the  discussions  and 
provide  insight  into  the  established  program  or  project 
goals. 

Participants  at  the  first  meeting  also  include  the  SRE 
team  leader  and  other  team  representatives. 


Focus 


The  focus  is  to  understand  the  client  program  or  project 
goals  and  to  determine  whether  the  SRE  can  meet 
those  goals. 

Table  5-1  is  a  set  of  suggested  topics  and  outcomes  for 
the  first  client  meeting. 

Table  5.1 :  Focus  of  First  Client  Meeting 


Topic  Outcome  ■■■ ; :  . 

Introductions 

•  Participants  are  introduced  and  understand  each  oth¬ 
er’s  functions. 

•  People  know  what  the  agenda  for  the  meeting  is. 

Briefing 

•  Client  has  good  understanding  of  the  SRE  purpose, 
scope,  products,  services,  and  research. 

•  Client  begins  to  formulate  a  context  for  the  SRE. 

Discussion: 

Client  Context 

SRE  team  has  good  understanding  of  the  strategic  con¬ 
text  for  the  evaluation: 

•  What  strategic  issues  are  facing  the  organization? 

•  What  are  the  long-term  objectives  associated  with  the 
completion  of  the  program? 

•  What  outcomes  are  expected  from  the  software  risk 
evaluation? 
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Table  5.1:  Focus  of  First  Client  Meeting  (continued) 


Topic 


Outcome 


Discussion: 
Client  Goals 


Business 

Relationship 


Closing 


SRE  team  understands  the  client’s  goals  for  the  effort: 

•  What  are  the  short-term  objectives  for  the  organization? 

•  How  critical  is  the  need? 

•  What  risks  are  foreseen  for  the  program  from  the  man¬ 
agers’  perspective? 

•  What  are  the  time  frames? 

•  What  kind  of  actions  is  management  prepared  to  take? 


SRE  team  and  client  understand  the  feasibility  of  per¬ 
forming  a  software  risk  evaluation. 

Both  parties  are  ready  to  do  business. 

The  SRE  team  understands  the  risk  evaluation’s  pur¬ 
pose  in  the  context  of  the  client’s  needs  and  goals. 
Need  is  determined  for  linkage  of  client’s  (sponsor/- 
manager)  goals  with  senior  management. 


•  Oudine  of  next  steps  to  be  taken  by  the  respective  indi¬ 
viduals. 

•  Summary  of  meeting  and  action  items. 

•  Closing  of  meeting. 


Follow-Up  Action  The  SRE  team  leader  must  follow-up  to  ensure  that  the 

business  aspect  of  the  relationship  between  the  client 
and  the  SRE  team  is  established  at  this  point. 
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Documenting 

Agreements 


Target  Project 
Acceptance 


Purpose 


5.1.2  Client  Agreement 


One  of  the  techniques  to  clarify  the  agreement  for  the 
risk  evaluation  and  to  ensure  its  understanding  by  the 
client  and  the  SRE  team  is  to  document  the  agreement 
and  have  both  parties  sign  it.  For  example,  the  SEI 
uses  a  Technical  Objectives  and  Plan  (TO&P)  docu¬ 
ment  that  is  formally  approved  by  both  the  SEI  man¬ 
agement  and  client  management  as  the  basis  for  con¬ 
ducting  the  risk  evaluations. 

The  client  program  or  project  goals  are  documented 
and  the  goals  that  are  identified  are  mapped  to  the  soft¬ 
ware  risk  taxonomy.  This  activity  is  necessary  to  focus 
the  software  risk  evaluation  on  the  goals. 


It  is  also  important  to  obtain  proper  acceptance  from 
the  management  and  other  executives  of  the  program 
or  project.  The  SRE  team  leader  should  ensure  that  the 
client  has  communicated  the  decision  to  perform  the 
risk  evaluation  to  the  proper  executives  and  that  they 
have  been  informed  well  in  advance. 


5.1.3  Second  Client  Meeting 


The  purpose  of  the  second  meeting  is  to  obtain  closure 
and  agreements  for  the  Software  Risk  Evaluation. 

In  addition  to  the  individuals  from  the  first  meeting, 
technical  focus  group(s)  from  the  client  organization 
may  also  participate  at  this  meeting. 
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Focus  The  focus  of  this  meeting  is  to  address  any  issues  that 

the  client  may  have  and  to  review  the  already  estab¬ 
lished  client  program  or  project  goals  for  the  Software 
Risk  Evaluation. 

This  meeting  should  also  address  the  logistics  and  spe¬ 
cific  resources  that  are  required  for  the  evaluation. 

Table  5-2  is  a  set  of  suggested  topics  and  outcomes  for 
the  second  client  meeting. 


Table  5.2:  Focus  of  Second  Client  Meeting 


Topic  * 

Outcome  ^72'. 

Introductions  and 
Meeting  Context 

•  Some  comfort  level  and  relationships  are  established. 

•  Purpose  of  meeting  -  understanding  the  SRE,  client’s 
profile,  context  for  the  SRE,  and  management  frame  of 
reference. 

•  Context  for  modifying  the  SRE  processes  including  tai¬ 
loring  to  meet  the  client’s  goals. 

Review  of  SRE 

•  Client  understands  what  the  SRE  does,  how  long  it 
takes,  what  resources  are  required,  roles,  and  the  typi¬ 
cal  outcome  after  performing  an  SRE. 

•  Client  understands  the  confidentiality  of  the  data. 

Discussion: 

Client  Issues 

•  SRE  team  understands  the  client’s  concerns  related  to 
risks  of  the  program  or  project. 

•  SRE  team  addresses  the  client’s  concerns  regarding 
the  risk  evaluation. 
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Table  5.2:  Focus  of  Second  Client  Meeting  (continued) 


Topic 

Outcome 

Discussion: 

Client  Goals  and 
Context  of  SRE 

•  SRE  team  understands  what  the  client  wants  from  the 
evaluation. 

•  SRE  understands  the  client’s  context  in  terms  of  the 
organization’s  strategic  direction  and  goals. 

Logistics  of  SRE 

•  Client  and  SRE  team  agree  on  logistics  for  the  soft¬ 
ware  risk  evaluation:  dates,  resources  needed,  site 
coordinator,  etc. 

•  Client  understands  the  importance  of  the  site  coordina¬ 
tor  role. 

Summary  and 
Agreements 

•  Outstanding  issues  related  to  the  conduct  or  purpose 
of  the  SRE  are  addressed  or  identified. 

•  Agreements  made  during  the  meeting,  dates  for  the 
evaluation,  site  coordinator,  etc.,  are  summarized. 

•  Date,  time,  and  personnel  identified  for  the  next  step 
are  established. 
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Introduction 


Purpose 


5.2  Preparation 


The  preparation  phase  consists  of  tailoring  the  risk 
evaluation,  selecting  the  SRE  team  and  interview 
group,  and  scheduling  of  activities.  The  preparation  ac¬ 
tivities  are  performed  by  the  SRE  team  leader  and  the 
site  coordinator. 

Figure  5-3  shows  the  sequence  of  activities  for  the 
preparation  phase. 


Figure  5-3:  Preparation  Activities 


5.2.1  Tailoring  the  SRE 


The  Software  Risk  Evaluation  is  tailored  to  the  client 
program  or  project  profile.  Tailoring  ensures  that  the 
SRE  will  meet  the  client  program  or  project  goals  for 
performing  the  risk  evaluation. 
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Specifics 


Team  Composition 


The  tailoring  activity  also  ensures  that  the  implementa¬ 
tion  details  of  the  SRE  method  can  be  effectively  ap¬ 
plied  to  the  program  or  project  specific  environment 
and  life-cycle  stage. 


The  tailoring  activity  makes  use  of  the  map  of  the  cli¬ 
ent's  goals  to  the  risk  taxonomy  that  was  developed 
earlier  during  the  commitment  phase. 

The  tailoring  activity  includes  identifying  the  type  and 
number  of  groups  that  should  be  interviewed  for  the 
SRE,  and  tailoring  the  Taxonomy-Based  Questionnaire 
to  meet  the  goals  of  the  client  program  or  project. 


5.2.2  SRE  Team  Selection 


The  Software  Risk  Evaluation  team  is  composed  of 
three  or  four  individuals  from  an  independent  organiza¬ 
tion  and  two  or  three  individuals  from  the  client  organi¬ 
zation  who  are  not  directly  associated  with  the  target 
program  or  project. 

The  client  members  of  the  team  should  not  have  a  re¬ 
porting  relationship  with  the  people  who  are  being  in¬ 
terviewed.  The  client  members  should  not  be  involved 
with  routine  activities  of  the  program  or  project  but 
should  have  a  knowledge  of  its  organization  and  tech¬ 
nical  and  business  aspects. 
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Team  Skills 


Team  Model 


Team  Selection 


SRE  team  members  should  possess  good  communica¬ 
tion  and  interpersonal  skills. 

Jointly,  the  team  must  have  a  working  knowledge  in  the 
areas  of  software  development  processes,  the  technol¬ 
ogy  used  on  the  project,  the  application  domain,  and 
relevant  organizational  procedures. 

The  team  should  have  undergone  a  formal  training 
course  prior  to  executing  the  Software  Risk  Evaluation. 


Ideally,  the  SRE  team  should  function  as  a  structured 

open  team  as  described  in  Constantine  [Constantine 

93].  This  involves  both  collaborative  team  work  and 

consensus  engineering. 

Some  of  the  special  features  in  the  team  model  are: 

•  A  catalog  of  essential  team  roles  is  identified. 

•  Formal  specification  of  and  institutionalization 
of  functional  roles  is  given. 

•  Rotation  of  roles  is  practiced  to  promote  flexi¬ 
bility  and  skill  acquisition. 

•  Default  assignment  of  roles  should  be  made  to 
assure  essential  functions  are  performed. 

•  A  structured  and  organized  record  of  the  group 
decisions  and  rationale  is  necessary. 

•Technical  consensus  building  is  more  impor¬ 
tant  than  majority-oriented  decision  making. 


The  SRE  team  leader  is  responsible  for  forming  the 
team  and  for  assigning  specific  roles  to  individuals  dur¬ 
ing  the  execution  phase. 
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Interview  Groups 


Functional  Groups 


Selection  Guidelines 


5.2.3  Interview  Group  Selection 

The  SRE  team  conducts  interviews  of  many  different 
groups  associated  with  the  program  or  project.  The  in¬ 
terview  groups  may  be  formed  by  organization,  region, 
products,  customers  and  contractors,  or  software  de¬ 
velopment  and  acquisition  functions. 
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The  following  are  some  of  the  typical  functional  groups 

that  may  be  represented  during  a  risk  evaluation: 

•  Software  designers,  developers,  and  testers 

•  Functional  support  personnel  such  as  inde¬ 
pendent  verification  &  validation  (IV&V),  quali¬ 
ty  assurance  (QA),  configuration  management 
(CM) 

•  Technical  leaders 

•  Systems  and  software  engineers 

•  Customers  and  users 

•  Contractors 

•  Project  managers,  functional  support  manag¬ 
ers,  and  program  managers 


The  individuals  for  the  interview  groups  are  selected 
based  on  both  their  knowledge  of  the  function  they  per¬ 
form  and  the  environment  of  the  target  program  or 
project. 

Ideally,  individuals  who  are  selected  are  willing  and 
able  to  express  themselves  in  a  group  setting.  Some¬ 
times,  however,  they  are  not.  It  is  the  responsibility  of 
the  SRE  team  to  ensure  the  responses. 
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The  individuals  within  each  interview  group  should  not 
have  any  direct  reporting  relationships  among  them¬ 
selves.  This  is  to  ensure  that  they  are  in  an  environ¬ 
ment  where  they  can  express  their  risks,  issues,  and 
concerns  without  any  fear  of  attribution. 


Group  Size  The  interview  groups  usually  consist  of  five  or  six  indi¬ 

viduals  for  various  reasons  such  as  to: 

•  manage  the  dynamics  of  a  group  interview  by 
the  interviewer 

•  provide  sufficient  time  for  follow-up  questions 

•  ensure  adequate  coverage  of  the  Taxonomy- 
Based  Questionnaire 

•  effectively  observe  and  engage  all  respon¬ 
dents  by  the  SRE  team 


Selection  The  site  coordinator  working  under  the  guidance  of  the 

SRE  team  leader  is  responsible  for  identifying  and 
scheduling  individuals  for  each  interview  session. 

The  site  coordinator  and  SRE  team  leader  are  also  re¬ 
sponsible  for  ensuring  that  interview  groups  are  select¬ 
ed  using  the  guidelines  of  the  SRE  method. 


5.2.4  Schedule  of  Activities 


Introduction  The  schedule  of  activities  for  the  SRE  team’s  on-site 

activities  is  planned  sufficiently  in  advance  so  that  indi¬ 
viduals  can  attend  the  sessions. 


46 


CMU/SEI-94-TR-1 9 


Implementation  Phases 


Although  some  of  the  sessions  must  be  completed  be¬ 
fore  others  can  begin,  the  schedule  can  be  adapted  to 
meet  the  needs  of  the  program  or  project.  The  site  co¬ 
ordinator  working  jointly  with  the  SRE  team  leader  es¬ 
tablishes  the  schedule.  The  site  coordinator  also  en¬ 
sures  that  rooms  for  the  sessions  are  scheduled  and 
communicated  to  the  participants. 


Typical  Schedule  Figure  5-4  shows  an  example  of  a  typical  SRE  sched¬ 
ule  of  activities. 


Day-1  Day-2  Day-3 

08:00  - : - - 1, - 

Site 

Orientation 

-  Interview  Interview 

09:00  -  project  I  Session  Session 

Overview  #3 


10:00 


Day-4  Day-5 


Interview 

Session 

#5 


Briefing 

Preparation 


|  Team 

11:00  I  Training 

12:00 


Analysis  I  _  P313  . 
Session  i  Confirmation 
#5  I  Briefing 


13:00 


14:00 


(Team 

Training 


16:00 


17:00 


Interview  Interview 
Session  Session 

#2  #4 


SRE  team  session 


Figure  5-4:  Typical  Schedule  of  Activities 
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Introduction 


Purpose 


5.3  Execution 


The  execution  phase  consists  of  those  activities  that 
are  performed  during  a  site  visit  of  the  SRE  team  to  the 
location  of  the  target  program  or  project. 

Figure  5-5  shows  the  sequence  of  activities  in  this 
phase. 


Figure  5-5:  Execution  Execution  Activities 


5.3.1  Site  Orientation 


The  purpose  of  the  site  orientation  is  for  all  individuals 
participating  in  the  execution  phase  activities  to  be 
aware  of: 

•the  objectives  for  performing  the  evaluation 

•the  implementation  phases  of  the  SRE 

•the  individual’s  role  in  the  implementation 
phases 
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Purpose 


Participants 


This  session  is  for  all  participants  who  are  involved  in 
the  execution  phase  activities  including  management 
and  technical  personnel  associated  with  the  program 
or  project  and  the  SRE  team. 

The  SRE  team  leader  is  responsible  for  conducting  the 
site  orientation. 


5.3.2  Program  Overview  Session 


The  purpose  of  the  overview  session  is  for  knowledge¬ 
able  representatives  to  present  in  summary  the  organi¬ 
zation’s  structure,  context,  and  the  technical  aspects  of 
the  target  program  or  project. 

The  objective  is  for  the  SRE  team  to  review  the  organi¬ 
zational  structure  and  the  characteristics,  terminology, 
and  functional  aspects  of  the  target  software  develop¬ 
ment  project. 


This  session  is  open  to  all  participants  involved  in  the 
risk  evaluation  including  management  and  technical 
personnel  associated  with  the  target  project. 
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5.3.3  SRE  Team  Training 


The  purpose  of  the  SRE  team  training  is  for  all  mem¬ 
bers  to  have  a  practical  knowledge  of  the  SRE  method 
prior  to  its  execution. 

Training  areas  include  the  SEI  risk  paradigm,  taxono¬ 
my  of  software  development  risks,  and  the  Taxonomy- 
Based  Questionnaire. 


The  training  also  provides  skills  required  to  perform  the 
risk  evaluation  functions  and  its  implementation  specif¬ 
ics. 

For  example,  the  training  prepares  the  team  in  the  use 
of  interviewing  techniques  and  risk  identification  skills 
using  simulated  scenarios  and  role  playing  exercises. 
The  exercises  give  team  members  the  practice  re¬ 
quired  for  interviewing  and  exposes  them  to  some  of 
the  typical  behaviors  that  can  be  expected  during  the 
actual  sessions. 


5.3.4  Interview  Sessions 


The  purpose  of  the  interview  sessions  is  to  perform  risk 
detection  and  specification. 


Each  interview  session  begins  with  the  interviewer  us¬ 
ing  an  introductory  script  followed  by  questions  from 
the  Taxonomy-Based  Questionnaire  (TBQ). 
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The  process  is  designed  to  facilitate  the  detection  of 
risks  on  the  basis  of  the  participants’  discussion  rather 
than  by  rigidly  following  the  structure  of  the  TBQ.  The 
participants  are  encouraged  to  follow  any  thread  of  dis¬ 
cussion  or  thought  as  long  as  they  are  objectively  dis¬ 
cussing  potential  risks  when  responding  to  the  ques¬ 
tions,  or  to  subsequent  follow-up  probes,  or  to  cues,  or 
begin  to  discuss  among  themselves. 


When  a  risk  is  identified  the  risk  recorder  will  document 
the  risk. 

The  risk  recorder  will  use  the  wording  of  the  respon¬ 
dents,  and  if  the  meaning  is  not  clear  will  clarify  the 
statement  before  documenting  it. 

The  risk  recorder  or  another  designated  person  is  re¬ 
sponsible  for  entering  the  risks  into  an  automated  anal¬ 
ysis  tool. 


A  session  recorder  takes  notes  on  the  context  and  oth¬ 
er  pertinent  discussions  during  the  risk  detection  activ¬ 
ities. 

Other  SRE  team  members  also  take  notes  to  ensure 
the  following: 

•  Any  potential  risk,  issue,  or  concern  that  was 
raised  by  an  interviewee  is  not  overlooked. 
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•  The  source  of  the  risk  is  clearly  identifiable  and 
can  be  tagged  to  a  category  in  the  SEI  taxon¬ 
omy  of  software  development  risks. 

•  Sufficient  information  is  available  for  the  team 
to  make  an  objective  assessment  of  each  risk 
that  was  detected. 


5.3.5  Analysis  Sessions 


The  purpose  of  the  analysis  session  is  to  complete  the 
functions  of  risk  specification  and  risk  assessment. 

The  analysis  session  is  performed  by  the  SRE  team. 
This  session  is  performed  immediately  after  each  inter¬ 
view  session  when  the  context  of  the  risk  is  still  fresh  in 
the  minds  of  the  SRE  team  members. 


Each  team  member  is  provided  a  copy  of  the  recorded 
risks.  The  SRE  team  discusses  only  those  risks  where 
the  wording  of  the  risk  statements  is  of  concern  and 
may  need  to  be  changed. 

The  team  members  individually  score  each  risk  using 
the  selected  assessment  mechanism.  The  team  then 
discusses  those  risks  that  have  a  significant  deviation 
in  their  scores  and  reaches  consensus. 

During  the  analysis  session  each  risk  will  also  be 
tagged  to  a  taxonomy  group.  That  is,  it  will  be  tagged 
as  belonging  to  a  specific  class-element-attribute  in  the 
software  risk  taxonomy. 
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During  this  session  the  risk  recorder  or  a  designated 
person  enters  the  risk  statement  corrections,  risk  as¬ 
sessment  scores,  and  source  of  risk  categories  into  the 
automated  analysis  tool. 

A  session  recorder  documents  the  team’s  decisions 
and  rationale  throughout  the  analysis  session. 


5.3.6  Consolidation  Session 


The  purpose  of  this  session  is  to  perform  the  risk  con¬ 
solidation  and  if  necessary,  revise  the  assessments  of 
the  consolidated  risks. 

This  session  is  performed  by  the  SRE  team  and  is  held 
after  all  the  analysis  sessions  have  been  completed. 


Each  SRE  team  member  is  provided  with  a  copy  of  the 
risks  from  the  analysis  sessions  sorted  by  their  levels 
of  magnitude  within  each  risk  category. 

The  SRE  team  jointly  examine  the  risks  within  each 
category  to  determine  if  there  are  candidates  for  con¬ 
solidation.  The  team  reaches  consensus  on  the  word¬ 
ing  and  revisits  the  assessment  scores  of  the  consoli¬ 
dated  risks  if  necessary. 
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A  risk  recorder  enters  the  consolidated  risk  statements 
and  their  revised  risk  assessments  into  the  automated 
analysis  tool. 

A  session  recorder  is  responsible  for  taking  notes  of  the 
session  and  documenting  the  team’s  decisions  and  ra¬ 
tionale  for  them. 


5.3.7  Briefing  Preparation 


The  purpose  of  this  session  is  to  prepare  for  the  data 
confirmation  briefing  to  be  presented  at  the  site. 


The  contents  of  the  briefing  are  a  listing  of  the  risk 
statements  that  were  identified  during  the  execution 
phase  and  sorted  by  their  source  of  risk  categories  and 
levels  of  magnitude. 

Although  all  risk  findings  may  be  presented,  the  focus 
of  the  data  confirmation  briefing  should  be  on  the  more 
important  ones,  that  is,  on  those  risks  that  were  as¬ 
sessed  at  a  high  level  of  magnitude. 


The  briefing  preparation  includes  a  review  of  the  con¬ 
tents  of  the  slides  that  will  be  presented.  This  is  done 
to  ensure  accuracy  and  verify  non-attribution  and  con¬ 
fidentiality  of  the  source  of  data. 

The  person  responsible  for  the  briefing  should  make 
the  necessary  corrections  to  the  slides. 
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5.3.8  Data  Confirmation  Briefing 


The  purpose  of  the  data  confirmation  briefing  is  to  vali¬ 
date  the  risk  findings  with  the  organization’s  manage¬ 
ment  and  all  individuals  who  participated  in  the  execu¬ 
tion  phase  activities. 

All  individuals  who  were  involved  in  the  execution 
phase  activities  including  the  management  and  techni¬ 
cal  personnel  associated  with  the  target  program  or 
project  and  the  SRE  team  members  are  encouraged  to 
attend  the  briefing  session. 

The  briefing  session  provides  feedback  to  the  organi¬ 
zation  by  openly  communicating  the  risks  that  were 
found  and  provides  an  opportunity  for  the  data  gath¬ 
ered  at  the  site  to  be  validated. 


After  the  briefing  a  copy  of  the  slides  that  were  present¬ 
ed  is  provided  to  the  target  organization’s  manage¬ 
ment. 

Any  errors  on  the  slides  are  communicated  to  the  team 
leader  who  ensures  the  corrections  are  made  in  the 
SRE  records. 
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5.4  Mitigation 


The  mitigation  phase  consists  of  activities  that  are  per¬ 
formed  to  develop  the  final  report  for  the  client. 

The  mitigation  phase  activities  are  performed  jointly  by 
the  SRE  team  and  a  representative  group  of  individuals 
from  the  target  program  or  project. 

Typically  one  ortwo  site  visits  are  necessary  to  perform 
these  activities. 

Figure  5-6  shows  the  sequence  of  activities  in  this 
phase. 


Figure  5-6:  Mitigation  Phase  Activities 
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5.4.1  Preliminary  Report 


The  purpose  of  the  preliminary  report  is  to  present  the 
SRE  team’s  analysis  of  risk  findings  and  proposed 
strategies  for  mitigating  the  risks. 

The  preparation  of  the  preliminary  report  is  performed 
by  the  SRE  team  after  the  execution  phase  activities 
are  completed. 

If  there  are  multiple  evaluations  or  execution  phases  for 
the  program  or  project,  then  an  additional  consolidation 
session  is  performed  prior  to  developing  the  prelimi¬ 
nary  report. 


The  preliminary  report  contains  the  consolidated  risk 
findings,  mitigation  areas,  analysis  of  risk  findings, 
goals,  and  proposed  mitigation  strategies  or  general 
recommendations. 


The  SRE  team  works  with  representatives  from  the 
project  or  program  for  developing  the  preliminary  re¬ 
port. 

The  group  analyzes  the  consolidated  risk  findings  and 
identifies  the  risk  mitigation  areas  for  the  program  or 
project.  The  group  also  produces  a  map  of  the  risk  find¬ 
ings  for  their  respective  mitigation  areas  and  the  cli¬ 
ent’s  goals.  Proposed  strategies  or  general  recommen¬ 
dations  for  mitigating  the  risks  may  also  be  included  in 
the  preliminary  report. 
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Purpose  The  purpose  of  this  meeting  is  to  review  the  preliminary 

report  with  the  client’s  management  and  other  desig¬ 
nees  from  the  program  or  project. 

It  is  important  that  the  client  has  had  sufficient  time  to 
review  the  preliminary  report  and  is  prepared  to  dis¬ 
cuss  it  at  this  meeting. 

Table  5-3  is  a  set  of  suggested  topics  and  outcome  for 
the  client  review  meeting. 


Table  5.3:  Focus  of  Client  Review  Meeting 


Topic 

Outcome 

Introductions 

•  Participants  are  introduced. 

•  Participants  know  what  the  agenda  for  the  meeting  is. 

•  Client  is  prepared  to  discuss  the  preliminary  report. 

Process  Overview 

•  Client  has  good  understanding  of  the  mitigation  pro¬ 
cess. 

•  Client  understands  the  preliminary  report  is  presented 
for  review  and  discussion  purposes. 

Discussion: 

Client  Context 

Preliminary  report  is  discussed  in  terms  of: 

•  Does  it  address  the  issues/risks  facing  the  organiza¬ 
tion? 

•  Does  it  address  the  objectives  for  successful  comple¬ 
tion  of  the  program  or  project? 

•  Does  it  meet  the  expectations/goals  that  were  set  at 
the  beginning  of  the  risk  evaluation? 
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Table  5.3:  Focus  of  Client  Review  Meeting  (continued) 


Topic 

Outcome 

Discussion: 

Mitigation  Areas/ 
Recommendations 

Preliminary  report  is  discussed  in  terms  of  the  mitigation 
areas  that  were  identified  and  proposed  or  general  rec¬ 
ommendations  if  any: 

•  Are  the  mitigation  areas  that  were  identified  linked  to 
the  goals  and  objectives  of  the  organization? 

•  Are  the  recommendations  practical  in  the  context  of  the 
client’s  goals  and  objectives? 

•  Are  there  any  issues  that  are  outstanding  or  have  not 
been  addressed? 

Discussion: 

Priorities 

•  What  priorities  would  the  client  assign  to  the  recom¬ 
mendations?  What  are  the  resources  that  are  avail¬ 
able?  What  are  the  time  frames? 

•  What  kind  of  actions  is  management  prepared  to  take? 

Summary 

•  Summarize  the  discussions  and  agreements. 

•  Outline  the  process  for  mitigation  activity  development. 

•  Identify  date,  time,  and  people  for  the  next  step. 

5.4.3  Mitigation  Activity  Development 


Purpose  The  purpose  of  the  mitigation  activity  development  is  to 

facilitate  the  process  of  developing  a  risk  mitigation 
plan  for  the  program  or  project.  The  process  includes 
identifying  recommendations  and  activities  as  well  as 
planning  details  for  mitigating  the  risks. 
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Mitigation  Plan  The  mitigation  activity  development  should  be  of  suffi¬ 
cient  detail  to  form  the  basis  for  implementing  the  miti¬ 
gation  activities  within  the  program  or  project.  In  this  re¬ 
gard  it  will  identify  the  sequence  of  activities,  costs  and 
benefits,  and  performance  measures.  The  client  man¬ 
agement  can  then  decide  who  should  perform  the  ac¬ 
tivities,  what  resources  should  be  allocated,  and  when 
each  activity  should  be  completed. 


Participants  The  mitigation  activity  development  requires  program- 

or  project-specific  knowledge  that  is  necessary  to  en¬ 
sure  the  mitigation  activities  are  practical. 

In  addition  to  the  program  or  project  representatives 
who  participated  in  producing  the  preliminary  report, 
other  individuals  who  can  provide  the  required  techni¬ 
cal  or  business  knowledge  may  also  be  required  to  par¬ 
ticipate. 


Logistics  The  mitigation  activity  development  may  be  completed 

in  a  single  2-  or  3-day  working  session,  or  spread  out 
over  many  meetings  of  shorter  durations. 

Although  the  meetings  can  be  held  at  any  location,  usu¬ 
ally  the  SRE  team  and  other  representatives  visit  the 
site  of  the  program  or  project.  This  is  not  only  conve¬ 
nient  for  referencing  materials  from  the  program  or 
project  but  also  for  inviting  other  knowledgeable  mem¬ 
bers  from  the  site,  if  the  need  arises. 
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For  each  mitigation  area  the  group  will  discuss  and 
identify  the  specific  recommendation  and  the  primary 
activities  that  are  required  to  be  performed.  The  group 
will  also  identify  constraints  within  the  program  or 
project  environment  that  are  barriers  for  implementing 
the  primary  activities. 

When  there  are  constraints  the  group  will  “brainstorm” 
to  identify  those  activities  that  should  be  performed  to 
overcome  the  barriers.  The  primary  activities  and  those 
that  are  required  to  overcome  any  constraints  are 
termed  key  activities  in  the  SRE  method. 

In  addition  the  group  will  identify  other  activities  that  are 
required  for  effectively  implementing  the 
recommendation.  Some  examples  of  the  other 
activities  are  producing  task  plans  or  conducting  kick¬ 
off  meetings.  These  activities  are  termed  enabling 
activities. 

The  process  of  mitigation  activity  development  also 
includes  sequencing  the  activities,  developing  costs 
and  benefits,  identifying  performance  measures,  and 
other  details  necessary  to  facilitate  the  mitigation  plan 
for  the  program  or  project. 
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5.4.4  Final  Report 


Purpose  The  purpose  of  the  final  report  is  to  provide  the  results 

of  the  risk  evaluation  to  the  management  of  the  client 
program  or  project. 


Contents  The  final  report  contains  an  executive  summary,  the 

general  recommendations  or  strategies  for  risk 
mitigation,  and  the  specific  mitigation  activities  that  are 
necessary  to  address  the  risks  that  were  identified  for 
the  program  or  project. 

The  contents  of  the  final  report  will  facilitate  the 
completion  of  the  mitigation  plan  that  can  be  used  for 
mitigating  the  risks  of  the  program  or  project. 

The  final  report  includes  other  topics  that  were 
provided  in  the  preliminary  report  with  the  modifications 
based  on  client  review.  These  are  risk  findings, 
mitigation  areas  and  analysis  of  findings,  mapping  of 
risks  to  mitigation  areas,  and  target  goals  for  each 
mitigation  area. 

A  copy  of  all  briefing  slides  and  other  materials  that 
were  used  in  the  risk  evaluation  are  also  provided  as  an 
appendix  to  the  final  report. 
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The  use  of  automated  tools  to  support  the  risk  evalua¬ 
tion  activities  is  recommended  as  it  ensures  that  the 
functions  will  be  performed  efficiently. 

Some  basic  tool  support  is  essential  for  completing  the 
tasks  within  the  limited  resources  and  time  that  are 
available  to  the  SRE  team  during  the  execution  phase. 


In  the  example  shown  in  Figure  6-1 ,  risk  statements 
that  are  identified  after  each  interview  session  are  en¬ 
tered  into  the  basic  tool.  This  exercise  is  done  prior  to 
the  beginning  of  each  analysis  session.  The  basic  tool 
is  then  used  to  print  the  analysis  sheets  that  will  be 
used  during  the  analysis  session. 

Later,  after  the  analysis,  the  results  of  the  session  are 
entered  into  the  tool  and  it  is  used  to  calculate  the  ag¬ 
gregate  scores  and  level  of  magnitude  of  the  risks. 


As  shown  in  Figure  6-1 ,  before  the  consolidation 
session  the  tool  is  used  to  print  a  complete  listing  of  the 
risks,  sorted  by  their  source  of  risk  and  level  of  magni¬ 
tude. 

This  listing  is  used  by  the  SRE  team  during  the  consol¬ 
idation  session. 
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Printed  list  of  risks  with  blank 
columns  for  assigning  source 
of  risk  &  assessment  scores; 
used  during  analysis  session 


P 


Printed  list  of  risks  sorted  by 
their  level  of  magnitude  for  each 
source  of  risk  category;  used 
during  consolidation  session 


P 


Printed  list  of  risks  sorted  by 
their  level  of  magnitude  for  each 
source  of  risk  category;  used  for 
for  preparing  the  briefing 


Printed  list  of  risk  and  source  of 
risk  for  each  magnitude  level; 
used  for  preparing  the  briefing 


Presentation  materials; 
used  for  the  data  confirmation 
briefing 


Figure  6-1  Use  of  Basic  Tools 


After  the  consolidation  session  any  changes  are 
entered  into  the  basic  tool.  The  tool  is  then  used  to  print 
out  reports  and  other  presentation  materials  that  are 
necessary  for  the  data  confirmation  briefing. 
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7  Post-Evaluation  Activities 

7.1  improvement  and  Enhancement 


After  each  risk  evaluation,  the  team  meets  to  discuss 
its  experience  and  the  lessons  learned  during  imple¬ 
mentation.  The  purpose  of  this  exercise  is  to  identify 
improvement  opportunities  for  the  SRE  method.  Modi¬ 
fications  to  the  existing  activities  and  any  new  require¬ 
ments  are  processed  through  a  formal  change  control 
mechanism  for  implementation.  This  ensures  continu¬ 
ous  improvement  and  enhancement  to  the  SRE  meth¬ 
od. 


7.2  Knowledge  Engineering 


The  SRE  results  are  normalized  and  stored  in  a 
database  containing  historical  risk  data.  This  is  per¬ 
formed  by  ensuring  the  data  will  maintain  the  confiden¬ 
tiality  of  the  program  or  project.  Any  proprietary  trade 
information  of  the  client  organization  will  not  be  stored 
in  the  database.  The  data  are  also  rendered  non-attrib¬ 
uting,  for  example,  by  removing  any  project-specific 
references  in  the  risk  statements. 


Domains  such  as  characteristics  of  the  project  or 
program,  the  structure  and  interactions  among  the 
technical  risks  that  were  identified,  cognitive  and  intel¬ 
lectual  processes  must  be  documented  and  analyzed 
as  well. 
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Appendix  A: 

Taxonomy  of  Software  Risk 


This  appendix  provides  the  definitions  of  the  Software 
Risk  Taxonomy’s  class,  element,  and  attributes. 

Figure  A-1  shows  an  overview  of  the  taxonomy  of  soft¬ 
ware  risk. 


Appendix  A 


A  Product  Engineering 


Product  engineering  refers  to  the  system  engineering 
and  software  engineering  activities  involved  in  creating 
a  system  which  satisfies  specified  requirements  and, 
also,  customer  expectations.  These  activities  include 
system  and  software  requirements  analysis  and  speci¬ 
fication,  software  design  and  implementation,  integra¬ 
tion  of  hardware  and  software  components,  and  soft¬ 
ware  and  system  test. 

The  elements  of  this  class  cover  traditional  software 
engineering  activities.  They  comprise  those  technical 
factors  associated  with  the  deliverable  product  itself, 
independent  of  the  processes  or  tools  used  to  produce 
it  or  the  constraints  imposed  by  finite  resources  or  ex¬ 
ternal  factors  beyond  program  control. 

Product  engineering  risks  generally  result  from  require¬ 
ments  which  are  technically  difficult  or  impossible  to  im¬ 
plement,  often  in  combination  with  inability  to  negotiate 
relaxed  requirements  or  revised  budgets  and  sched¬ 
ules;  from  inadequate  analysis  of  requirements  or  de¬ 
sign  specifications,  or  from  poor  quality  design  or  cod¬ 
ing  specifications. 


1  Requirements 


Attributes  of  the  requirements  element  cover  both  the 
quality  of  the  requirements  specification  and,  also,  the 
difficulty  of  implementing  a  system  which  satisfies  the 
requirements. 
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1  A.  Product  Engineering 

B.  Development 

C.  Program  Constraints 
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1. 
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c.  Interfaces 
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d.  Performance 
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3.  Management  Process 

d.  Prime 

g.  Non- 
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a.  Planning 

b.  Project 
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e.  Corporate 
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Code  and  Unit  Test 

c.  Management 
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f.  Specifications 

d.  Morale 

Figure  A-1  Taxonomy  of  Software  Risk  -  An  Overview 
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The  following  attributes  characterize  the  requirements 
element. 


a)  Stability 


The  stability  attribute  refers  to  the  degree  to  which  the 
requirements  are  changing  and  the  possible  effect 
changing  requirements  and  external  interfaces  will 
have  on  the  quality,  functionality,  schedule,  design,  in¬ 
tegration,  and  testing  of  the  product  being  built. 

The  attribute  also  includes  issues  which  arise  from  the 
inability  to  control  rapidly  changing  requirements.  For 
example,  impact  analyses  may  be  inaccurate  because 
it  is  impossible  to  define  the  baseline  against  which  the 
changes  will  be  implemented. 


b)  Completeness 


Missing  or  incompletely  specified  requirements  may 
appear  in  many  forms,  such  as:  a  requirements  docu¬ 
ment  with  many  functions  or  parameters  “to  be  de¬ 
fined,”  requirements  that  are  not  specified  adequately 
to  develop  acceptance  criteria,  or  inadvertently  omitted 
requirements.  When  missing  information  is  not  sup¬ 
plied  in  a  timely  manner,  implementation  may  be  based 
on  contractor  assumptions  which  differ  from  customer 
expectations. 
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When  customer  expectations  are  not  documented  in 
the  specification,  they  are  not  budgeted  into  the  cost 
and  schedule. 


c)  Clarity 


This  attribute  refers  to  ambiguously  or  imprecisely  writ¬ 
ten  individual  requirements  which  are  not  resolved  until 
late  in  the  development  phase.  This  lack  of  a  mutual 
contractor  and  customer  understanding  may  require 
re-work  to  meet  the  customer  intent  for  a  requirement. 


d)  Validity 


This  attribute  refers  to  whether  the  aggregate  require¬ 
ments  reflect  customer  intentions  for  the  product.  This 
may  be  affected  by  misunderstandings  of  the  written 
requirements  by  the  contractor  or  customer,  unwritten 
customer  expectations  or  requirements,  or  a  specifica¬ 
tion  in  which  the  end  user  did  not  have  inputs. 

This  attribute  is  affected  by  the  completeness  and  clar¬ 
ity  attributes  of  the  requirements  specifications,  but  re¬ 
fers  to  the  larger  question  of  the  system  as  a  whole 
meeting  customer  intent. 
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. . . aw . hi . mmmmmmsmmmmmmmsmmm 

e)  Feasibility 


The  feasibility  attribute  refers  to  the  difficulty  of  imple¬ 
menting  a  single  technical  or  operational  requirement, 
or  of  simultaneously  meeting  conflicting  requirements. 
Sometimes  two  requirements  are  by  themselves  are 
feasible,  but  together  are  not;  they  cannot  both  exist  in 
the  same  product  at  the  same  time. 

Also  included  is  the  ability  to  determine  an  adequate 
qualification  method  for  demonstration  that  the  system 
satisfies  the  requirement. 


f)  Precedent 


The  precedent  attribute  concerns  capabilities  that  have 
not  been  successfully  implemented  in  any  existing  sys¬ 
tems  or  are  beyond  the  experience  program  personnel 
or  of  the  company.  The  degree  of  risk  depends  on  allo¬ 
cation  of  additional  schedule  and  budget  to  determine 
the  feasibility  of  their  implementation;  contingency 
plans  in  case  the  requirements  are  not  feasible  as  stat¬ 
ed;  and  flexibility  in  the  contract  to  allocate  implemen¬ 
tation  budget  and  schedule  based  on  the  outcome  of 
the  feasibility  study. 

Even  when  unprecedented  requirements  are  feasible, 
there  may  still  be  a  risk  of  underestimating  the  difficulty 
of  implementation  and  committing  to  an  inadequate 
budget  and  schedule. 
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g)  Scale 


This  attribute  covers  both  technical  and  management 
challenges  presented  by  large  complex  systems  devel¬ 
opment. 

Technical  challenges  include  satisfaction  of  timing, 
scheduling  and  response  requirements,  communica¬ 
tion  among  processors,  complexity  of  system  integra¬ 
tion,  analysis  of  inter-component  dependencies,  and 
impacts  due  to  changes  in  requirements. 

Management  of  a  large  number  of  tasks  and  people  in¬ 
troduces  a  complexity  in  such  areas  as  project  organi¬ 
zation,  delegation  of  responsibilities,  communication 
among  management  and  peers,  and  configuration 
management. 


2  Design 


The  attributes  of  the  design  element  cover  the  design 
and  feasibility  of  algorithms,  functions  or  performance 
requirements  and  of  the  internal  and  external  product 
interfaces.  Difficulty  in  testing  may  begin  here  with  fail¬ 
ure  to  work  to  testable  requirements  or  to  include  test 
features  in  the  design. 

The  following  attributes  characterize  the  design  ele¬ 
ment. 


A-8 


CMU/SEI-94-TR-1 9 


Appendix  A 


a)  Functionality 


This  attribute  covers  functional  requirements  which 
may  not  submit  to  a  feasible  design,  or  use  of  specified 
algorithms  or  designs  without  a  high  degree  of  certainty 
that  they  will  satisfy  their  source  requirements.  Algo¬ 
rithm  and  design  studies  may  not  have  used  appropri¬ 
ate  investigation  techniques  or  may  show  marginal  fea¬ 
sibility. 


b)  Difficulty 


The  difficulty  attribute  refers  to  functional  or  design  re¬ 
quirements  that  may  be  extremely  difficult  to  realize. 
Systems  engineering  may  design  a  system  architec¬ 
ture  difficult  to  implement,  or  requirements  analysis 
may  have  been  based  on  optimistic  design  assump¬ 
tions. 

The  difficulty  attribute  differs  from  design  feasibility  in 
that  it  does  not  proceed  from  pre-ordained  algorithms 
or  designs. 


c)  interfaces 


This  attribute  covers  all  hardware  and  software  inter¬ 
faces  that  are  within  the  scope  of  the  development  pro¬ 
gram  including  interfaces  between  configuration  items, 
and  the  existence  of  techniques  for  defining  and  man¬ 
aging  the  interfaces. 
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Special  note  is  taken  of  non-developmental  software 
and  developmental  hardware  interfaces. 


d)  Performance 


The  performance  attribute  refers  to  all  aspects  of  per¬ 
formance:  user  and  real-time  response  requirements, 
throughput  requirements,  performance  analyses,  and 
performance  modeling  throughout  the  development  cy¬ 
cle. 


e)  Testability 


The  testability  attribute  covers  the  amenability  of  the 
design  to  testing,  design  of  features  to  facilitate  testing, 
and  the  inclusion  in  the  design  process  of  people  who 
will  design  and  conduct  product  tests. 


f)  Hardware  Constraints 


This  attribute  covers  target  hardware  with  respect  to 
system  and  processor  architecture,  and  the  depen¬ 
dence  on  hardware  to  meet  system  and  software  per¬ 
formance  requirements.  These  constraints  may  include 
throughput  or  memory  speeds,  real-time 
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response  capability,  database  access  or  capacity  limi¬ 
tations,  insufficient  reliability,  unsuitability  to  system 
function,  or  insufficiency  in  the  amount  of  specified 
hardware. 


g)  Non-Developmental  Software 


Since  non-developmental  software  (NDS)  is  not  de¬ 
signed  to  system  requirements,  but  selected  as  a  “best 
fit,”  it  may  not  conform  precisely  to  performance,  oper¬ 
ability  or  supportability  requirements. 

The  customer  may  not  accept  vendor  or  developer  test 
and  reliability  data  to  demonstrate  satisfaction  of  the 
requirements  allocated  to  NDS.  It  may  then  be  difficult 
to  produce  this  data  to  satisfy  acceptance  criteria  and 
only  within  the  estimated  NDS  test  budget. 

Requirements  changes  may  necessitate  reengineering 
or  reliance  on  vendors  for  special  purpose  upgrades. 


3  Code  and  Unit  Test 


Attributes  of  this  element  are  associated  with  the  qual¬ 
ity  and  stability  of  software  or  interface  specifications, 
and  the  constraints  which  may  present  implementation 
or  test  difficulties. 
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a)  Feasibility 


The  feasibility  attribute  of  the  code  and  unit  test  ele¬ 
ment  addresses  possible  difficulties  which  may  arise 
from  poor  design  or  design  specification  or  from  inher¬ 
ently  difficult  implementation  needs. 

For  example,  the  design  may  not  have  quality  attributes 
such  as  module  cohesiveness  or  interface  minimiza¬ 
tion;  the  size  of  the  modules  may  contribute  to  design 
complexity;  the  design  may  not  be  specified  in  suffi¬ 
cient  detail,  requiring  the  programmer  to  make  as¬ 
sumptions  or  design  decisions  during  coding;  or  the  de¬ 
sign  and  interface  specifications  may  be  changing,  per¬ 
haps  without  an  approved  detailed  design  baseline; 
and,  the  use  of  developmental  hardware  may  make  an 
additional  contribution  to  inadequate  or  unstable  inter¬ 
face  specification.  Or,  the  nature  of  the  system  itself 
may  aggravate  the  difficulty  and  complexity  of  the  cod¬ 
ing  task. 


b)  Unit  Test 


Factors  affecting  unit  test  include  planning  and  prepa¬ 
ration  and  also  the  resources  and  time  allocated  for 
test. 

Constituents  of  these  factors  are:  entering  unit  test  with 
quality  code  obtained  from  formal  or  informal  code  in¬ 
spection  or  verification  procedures;  pre-planned  test 
cases  which  have  been  verified  to  test  unit 
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requirements;  a  test  bed  consisting  of  the  necessary 
hardware  or  emulators,  and  software  or  simulators, 
and  test  data  to  satisfy  the  planned  test;  sufficient 
schedule  to  plan  and  carry  out  the  test  plan. 


c)  Coding/Implementation 


This  attribute  addresses  the  implications  of  implemen¬ 
tation  constraints.  Some  of  these  are:  target  hardware 
which  is  marginal  or  inadequate  with  regard  to  speed, 
architecture,  memory  size  or  external  storage  capacity; 
required  implementation  languages  or  methods;  or  dif¬ 
ferences  between  the  development  and  target  hard¬ 
ware. 


4  Integration  and  Test 


This  element  covers  integration  and  test  planning,  exe¬ 
cution  and  facilities  for  both  the  contractual  product  and 
for  the  integration  of  the  product  into  the  system  or  site 
environment. 


88888888888888888888888888888888888888^^ 

a)  Environment 


The  integration  and  test  environment  includes  the 
hardware  and  software  support  facilities  and  adequate 
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test  cases  reflecting  realistic  operational  scenarios, 
and  realistic  test  data  and  conditions. 

This  attribute  addresses  the  adequacy  of  this  environ¬ 
ment  to  enable  integration  in  a  realistic  environment  or 
to  fully  test  all  functional  and  performance  require¬ 
ments. 


b)  Product  Integration 


The  product  integration  attribute  refers  to  integration  of 
the  software  components  to  each  other  and  to  the  tar¬ 
get  hardware,  and  testing  of  the  contractually  deliver¬ 
able  product.  Factors  which  may  affect  this  are  internal 
interface  specifications  for  either  hardware  or  software, 
testability  of  requirements,  negotiation  of  customer 
agreement  on  test  criteria,  adequacy  of  test  specifica¬ 
tions,  and  sufficiency  of  time  for  integration  and  test. 


c)  System  Integration 


The  system  integration  attribute  refers  to  integration  of 
the  contractual  product  to  interfacing  systems  or  sites. 
Factors  associated  with  this  attribute  are  external  inter¬ 
face  specifications,  ability  to  faithfully  produce  system 
interface  conditions  prior  to  site  or  system  integration, 
access  to  the  system  or  site  being  interfaced  to,  ade¬ 
quacy  of  time  fortesting,  and  associate  contractor  rela¬ 
tionships. 
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5  Engineering  Specialities 


The  engineering  specialty  requirements  are  treated 
separately  from  the  general  requirements  element  pri¬ 
marily  because  they  are  often  addressed  by  specialists 
who  may  not  be  full-time  on  the  program.  This  taxo¬ 
nomic  separation  is  a  device  to  ensure  that  these  spe¬ 
cialists  are  called  in  to  analyze  the  risks  associated  with 
their  areas  of  expertise. 


a)  Maintainability 


Maintainability  may  be  impaired  by  poor  software  archi¬ 
tecture,  design,  code,  or  documentation  resulting  from 
undefined  or  un-enforced  standards  and  requirements, 
or  from  neglecting  to  analyze  the  system  from  a  main¬ 
tenance  point  of  view. 


b)  Reliability 


System  reliability  or  availability  requirements  may  be 
affected  by  hardware  not  meeting  its  reliability  specifi¬ 
cations  or  system  complexity  which  aggravates  difficul¬ 
ties  in  meeting  recovery  timeliness.  Reliability  or  avail¬ 
ability  requirements  allocated  to  software  may  be  stat¬ 
ed  in  absolute  terms,  rather  than  as  separable  from 
hardware  and  independently  testable. 
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c)  Safety 


This  attribute  addresses  the  difficulty  of  implementing 
allocated  safety  requirements  and  also  the  potential 
difficulty  of  demonstrating  satisfaction  of  requirements 
by  faithful  simulation  of  the  unsafe  conditions  and  cor¬ 
rective  actions.  Full  demonstration  may  not  be  possible 
until  the  system  is  installed  and  operational. 


d)  Security 


This  attribute  addresses  lack  of  experience  in  imple¬ 
menting  the  required  level  of  system  security  which 
may  result  in  under-estimation  of  the  effort  required  for 
rigorous  verification  methods,  certification  and  accred¬ 
itation,  and  secure  or  trusted  development  process  lo¬ 
gistics;  developing  to  unprecedented  requirements; 
and  dependencies  on  delivery  of  certified  hardware  or 
software. 


e)  Human  Factors 


Meeting  human  factors  requirements  is  dependent  on 
understanding  the  operational  environment  of  the  in¬ 
stalled  system  and  agreement  with  various  customer 
and  user  factions  on  a  mutual  understanding  of  the  ex¬ 
pectations  embodied  in  the  human  factors  require¬ 
ments. 
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It  is  difficult  to  convey  this  understanding  in  a  written 
specification.  Mutual  agreement  on  the  human  inter¬ 
face  may  require  continuous  prototyping  and  demon¬ 
stration  to  various  customer  factions. 


f)  Specifications 


This  attribute  addresses  specifications  for  the  system, 
hardware,  software,  interface,  or  test  requirements  or 
design  at  any  level  with  respect  to  feasibility  of  imple¬ 
mentation  and  the  quality  attributes  of  stability,  com¬ 
pleteness,  clarity  and  verifiability. 
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B  Development  Environment 


The  development  environment  class  of  the  Software 
Development  Risk  Taxonomy  addresses  the  project 
environment  and  the  process  used  to  engineer  a  soft¬ 
ware  product.  This  environment  includes  the  develop¬ 
ment  process  and  system,  management  methods,  and 
work  environment.  These  environmental  elements  are 
characterized  below  by  their  component  attributes. 


1  Development  Process 


The  development  process  element  refers  to  the  pro¬ 
cess  by  which  the  contractor  proposes  to  satisfy  the 
customer's  requirements.  The  process  is  the  sequence 
of  steps — the  inputs,  outputs,  actions,  validation  crite¬ 
ria,  and  monitoring  activities —  leading  from  the  initial 
requirement  specification  to  the  final  delivered  product. 
The  development  process  includes  such  phases  as  re¬ 
quirements  analysis,  product  definition,  product  cre¬ 
ation,  testing,  and  delivery.  It  includes  both  general 
management  processes  such  as  costing,  schedule 
tracking,  and  personnel  assignment,  and  also  project- 
specific  processes  such  as  feasibility  studies,  design 
reviews,  and  regression  testing. 

This  element  groups  risks  that  result  from  a  develop¬ 
ment  process  that  is  inadequately  planned,  defined 
and  documented;  that  is  not  suited  to  the  activities  nec¬ 
essary  to  accomplish  the  project  goals;  and  that  is 
poorly  communicated  to  the  staff  and  lacks  enforced 
usage. 
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a)  Formality 


Formality  of  the  development  process  is  a  function  of 
the  degree  to  which  a  consistent  process  is  defined, 
documented,  and  communicated  for  all  aspects  and 
phases  of  the  development. 


b)  Suitability 


Suitability  refers  to  the  adequacy  with  which  the  select¬ 
ed  development  model,  process,  methods  and  tools 
support  the  scope  and  type  of  activities  required  for  the 
specific  program. 


c)  Process  Control 


Process  control  refers  not  only  to  ensuring  usage  of  the 
defined  process  by  program  personnel,  but  also  to  the 
measurement  and  improvement  of  the  process  based 
on  observation  with  respect  to  quality  and  productivity 
goals.  Control  may  be  complicated  due  to  distributed 
development  sites. 
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d)  Familiarity 


Familiarity  with  the  development  process  covers 
knowledge  of,  experience  in,  and  comfort  with  the  pre¬ 
scribed  process. 


e)  Product  Control 


Product  control  is  dependent  on  traceability  of  require¬ 
ments  from  the  source  specification  through  implemen¬ 
tation,  such  that  the  product  test  will  demonstrate  the 
source  requirements.  The  change  control  process 
makes  use  of  the  traceability  mechanism  in  impact 
analyses  and  reflects  all  resultant  document  modifica¬ 
tions  including  interface  and  test  documentation. 


2  Development  System 


The  development  system  element  addresses  the  hard¬ 
ware  and  software  tools  and  supporting  equipment 
used  in  product  development.  This  includes  CASE 
tools,  simulators,  compilers,  test  equipment,  and  host 
computer  systems. 
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a)  Capacity 


Risks  associated  with  the  capacity  of  the  development 
system  may  result  from  too  few  workstations,  insuffi¬ 
cient  processing  power  or  database  storage,  or  other 
inadequacies  in  equipment  to  support  parallel  activities 
for  development,  test  and  support  activities. 


b)  Suitability 


Suitability  of  the  development  system  is  associated 
with  the  degree  to  which  it  is  supportive  of  the  specific 
development  models,  processes,  methods,  proce¬ 
dures  and  activities  required  and  selected  for  the  pro¬ 
gram.  This  includes  the  development,  management, 
documentation,  and  configuration  management  pro¬ 
cesses. 


c)  Usability 


Usability  refers  to  development  system  documentation, 
accessibility  and  workspace  as  well  as  ease  of  use. 
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d)  Familiarity 


Development  system  familiarity  depends  on  prior  use 
of  the  system  by  the  company  and  by  project  personnel 
as  well  as  adequate  training  for  new  users. 


e)  Reliability 


Development  system  reliability  is  a  measure  of  whether 
the  needed  components  of  the  development  system 
are  available  and  working  properly  whenever  required 
by  any  program  personnel. 


f)  System  Support 


Development  system  support  involves  training  in  use  of 
the  system,  access  to  expert  users  or  consultants,  and 
repair  or  resolution  of  problems  by  vendors. 


g)  Deliverability 


Some  contracts  require  delivery  of  the  development 
system.  Risks  may  result  from  neglecting  to  bid  and  al¬ 
locate  resources  to  ensure  that  the  development  sys¬ 
tem  meets  all  deliverable  requirements. 
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3  Management  Process 


The  management  process  element  pertains  to  risks  as¬ 
sociated  with  planning,  monitoring  and  controlling  bud¬ 
get  and  schedule;  with  controlling  factors  involved  in 
definition,  implementation,  and  test  of  the  product;  with 
management  of  project  personnel;  and  with  handling 
external  organizations  including  the  customer,  senior 
management,  matrix  management  and  other  contrac¬ 
tors. 


a)  Planning 


The  planning  attribute  addresses  risks  associated  with 
developing  a  well-defined  plan  which  is  responsive  to 
contingencies  as  well  as  long  range  goals  and  which 
was  formulated  with  the  input  and  acquiescence  of 
those  affected  by  it.  Also  addressed  are  managing  ac¬ 
cording  to  the  plan  and  formally  modifying  the  plan 
when  changes  are  necessary. 


b)  Project  Organization 


Mti 


This  attribute  addresses  the  effectiveness  of  the  pro¬ 
gram  organization,  the  effective  definition  of  roles  and 
responsibilities,  and  the  assurance  that  these  roles  and 
lines  of  authority  are  understood  by  program  person¬ 
nel. 
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c)  Management  Experience 


This  attribute  refers  to  the  experience  of  all  levels  of 
managers  with  respect  to  management,  software  de¬ 
velopment  management,  the  application  domain,  the 
scale  and  complexity  of  the  system  and  program,  the 
selected  development  process,  and  hands-on  develop¬ 
ment  of  software. 


d)  Program  interfaces 


This  attribute  refers  to  the  interactions  of  managers  at 
all  levels  with  program  personnel  at  all  levels,  and  with 
external  personnel  such  as  the  customer,  senior  man¬ 
agement,  and  peer  managers. 


4  Management  Methods 


This  element  refers  to  methods  for  managing  both  the 
development  of  the  product  and  program  personnel. 
These  include  quality  assurance,  configuration  man¬ 
agement,  staff  development  with  respect  to  program 
needs,  and  maintaining  communication  about  program 
status  and  needs. 
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a)  Monitoring 


The  monitoring  includes  the  activities  of  obtaining  and 
acting  upon  status  reports,  allocating  status  informa¬ 
tion  to  the  appropriate  program  organizations,  and 
maintaining  and  using  progress  metrics. 


b)  Personnel  Management 


Personnel  management  refers  to  selection  and  training 
of  program  members  and  ensuring  that  they:  take  part 
in  planning  and  customer  interaction  for  their  areas  of 
responsibility;  work  according  to  plan;  receive  the  help 
they  need  or  ask  for  to  carry  out  their  responsibilities. 


c)  Quality  Assurance 


The  quality  assurance  attribute  refers  to  the  proce¬ 
dures  instituted  for  ensuring  that  contractual  processes 
and  standards  are  implemented  properly  for  all  pro¬ 
gram  activities,  and  that  the  quality  assurance  function 
is  adequately  staffed  to  perform  their  duties. 
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d)  Configuration  Management 


The  configuration  management  (CM)  attribute  ad¬ 
dresses  both  staffing  and  tools  for  the  CM  function  as 
well  as  the  complexity  of  the  required  CM  process  with 
respect  to  such  factors  as  multiple  development  and  in¬ 
stallation  sites  and  product  coordination  with  existing, 
possibly  changing,  systems. 


5  Work  Environment 


The  work  environment  element  refers  to  subjective  as¬ 
pects  of  the  environment  such  as  the  amount  of  care 
given  to  ensuring  that  people  are  kept  informed  of  pro¬ 
gram  goals  and  information,  the  way  people  work  to¬ 
gether,  responsiveness  to  staff  inputs,  and  the  attitude 
and  morale  of  the  program  personnel. 


a)  Quality  Attitude 


This  attribute  refers  to  the  tendency  of  program  person¬ 
nel  to  do  quality  work  in  general  and  to  conform  to  spe¬ 
cific  quality  standards  for  the  program  and  product. 
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b)  Cooperation 


The  cooperation  attribute  addresses  lack  of  team  spirit 
among  development  staff  both  within  and  across  work 
groups  and  the  failure  of  all  management  levels  to 
demonstrate  that  best  efforts  are  being  made  to  re¬ 
move  barriers  to  efficient  accomplishment  of  work. 


c)  Communication 


Risks  which  result  from  poor  communication  are  due  to 
lack  of  knowledge  of  the  system  mission,  requirements 
and  design  goals  and  methods,  or  to  lack  of  information 
about  the  importance  of  program  goals  to  the  company 
or  the  project. 


d)  Morale 


Risks  that  result  from  low  morale  range  across  low  lev¬ 
els  of  enthusiasm  and  thus  low  performance,  produc¬ 
tivity  or  creativity;  anger  which  may  result  in  intentional 
damage  to  the  project  or  the  product;  mass  exodus  of 
staff  from  the  project;  and  a  reputation  within  the  com¬ 
pany  which  makes  it  difficult  to  recruit. 
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C  Program  Constraints 


Program  constraints  refer  to  the  “externals”  of  the 
project.  These  are  factors  that  may  be  outside  the  con¬ 
trol  of  the  project  but  can  still  have  major  effects  on  its 
success  or  constitute  sources  of  substantial  risk. 


1  Resources 


This  element  addresses  resources  for  which  the  pro¬ 
gram  is  dependent  on  factors  outside  program  control 
to  obtain  and  maintain.  These  include  schedule,  staff, 
budget,  and  facilities. 


a)  Schedule 


This  attribute  refers  to  the  stability  of  the  schedule  with 
respect  to  internal  and  external  events  or  dependen¬ 
cies  and  the  viability  of  estimates  and  planning  for  all 
phases  and  aspects  of  the  program. 


b)  Staff 


This  attribute  refers  to  the  stability  and  adequacy  of  the 
staff  in  terms  of  numbers  and  skill  levels,  their  experi¬ 
ence  and  skills  in  the  required  technical  areas 
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and  application  domain,  as  well  as  their  availability 
when  needed. 


c)  Budget 


This  attribute  refers  to  the  stability  of  the  budget  with  re¬ 
spect  to  internal  and  external  events  or  dependencies 
and  the  viability  of  estimates  and  planning  for  all  phas¬ 
es  and  aspects  of  the  program. 


d)  Facilities 


This  attribute  refers  to  the  adequacy  of  the  program  fa¬ 
cilities  for  development,  integration,  and  testing  of  the 
product. 


2  Contract 


Risks  associated  with  the  program  contract  are  classi¬ 
fied  according  to  contract  type,  restrictions,  and  depen¬ 
dencies. 
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a)  Type  of  contract 


This  attribute  covers  the  payment  terms  (cost  plus 
award  fee,  cost  plus  fixed  fee,...)  and  the  contractual 
requirements  associated  with  such  items  as  the  State¬ 
ment  of  Work,  Contract  Data  Requirements  List,  and 
the  amount  and  conditions  of  customer  involvement. 


b)  Restrictions 


Contract  restrictions  and  restraints  refer  to  contractual 
directives  to,  for  example,  use  specific  development 
methods,  standards,  or  equipment  and  the  resultant 
complications  such  as  acquisition  of  data  rights  for  use 
of  non-developmental  software. 


c)  Dependencies 


This  attribute  refers  to  the  possible  contractual  depen¬ 
dencies  on  outside  contractors  or  vendors,  customer 
furnished  equipment  or  software,  or  other  outside  prod¬ 
ucts  and  services. 


CMU/SEI-94-TR-1 9 


A-31 


Appendix  A 


3  Program  Interfaces 


M 


This  element  consists  of  the  various  interfaces  with 
entities  and  organizations  outside  the  development 
program  itself. 


a)  Customer 


The  customer  attribute  refers  to  the  customer’s  level  of 
skill  and  experience  in  the  technical  or  application  do¬ 
main  of  the  program  as  well  as  difficult  working  relation¬ 
ships  or  poor  mechanisms  for  attaining  customer 
agreement  and  approvals,  not  having  access  to  certain 
customer  factions,  or  not  being  able  to  communicate 
with  the  customer  in  a  forthright  manner. 


b)  Associate  Contractors 


The  presence  of  associate  contractors  may  introduce 
risks  due  to  conflicting  political  agenda,  problems  of  in¬ 
terfaces  to  systems  being  developed  by  outside  orga¬ 
nizations,  or  lack  of  cooperation  in  coordination  of 
schedules  and  configuration  changes. 


c)  Subcontractors 


The  presence  of  subcontractors  may  introduce  risks 
due  to  inadequate  task  definitions  and  subcontractor 
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management  mechanisms,  or  to  not  transferring  sub¬ 
contractor  technology  and  knowledge  to  the  program 
or  corporation. 


d)  Prime  Contractor 


When  the  program  is  a  subcontract,  risks  may  arise 
from  poorly  defined  task  definitions,  complex  reporting 
arrangements,  or  dependencies  on  technical  or  pro¬ 
grammatic  information. 


e)  Corporate  Management 


Risks  in  the  corporate  management  area  include  poor 
communication  and  direction  from  senior  management 
as  well  as  non-optimum  levels  of  support. 


f)  Vendors 


Vendor  risks  may  present  themselves  in  the  forms  of 
dependencies  on  deliveries  and  support  for  critical  sys¬ 
tem  components. 


g)  Politics 


Appendix  A 


Political  risks  may  accrue  from  relationships  with  the 
company,  customer,  associate  contractors  or  subcon¬ 
tractors  and  may  affect  technical  decisions.company, 
customer,  associate  contractors  or  subcontractors  and 
may  affect  technical  decisions. 
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A.  Product  Engineering 


1.  Requirements 

a)  Stability 

[Are  requirements  changing  even  as  the  product  is  being 
produced?] 

[1]  Are  the  requirements  stable? 

(No)  (l.a)  What  is  the  effect  on  the  system? 

•  Quality 

•  Functionality 

•  Schedule 

•  Integration 

•  Design 

•  Testing 

[2]  Are  the  external  interfaces  changing? 

b)  Completeness 

[Are  requirements  missing  or  incompletely  specified?] 

[3]  Are  there  any  TBDs  in  the  specifications? 

[4]  Are  there  requirements  you  know  should  be  in 
the  specification  but  aren’t? 

(Yes)  (4.a)  Will  you  be  able  to  get  these  re¬ 
quirements  into  the  system? 

[5]  Does  the  customer  have  unwritten  require¬ 
ments/expectations? 

(Yes)  (5.a)  Is  there  a  way  to  capture  these  re¬ 
quirements? 

[6]  Are  the  external  interfaces  completely  defined? 
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c)  Clarity 

[Are  requirements  unclear  or  in  need  of  interpretation?] 

[7]  Are  you  able  to  understand  the  requirements  as 
written? 

(No)  (7.a)  Are  the  ambiguities  being  resolved 
satisfactorily? 

(Yes)  (7.b)  There  are  no  ambiguities  or  prob¬ 
lems  of  interpretation? 

d)  Validity 

[Will  the  requirements  lead  to  the  product  the  customer 
has  in  mind?] 

[8]  Are  there  any  requirements  that  may  not  specify 
what  the  customer  really  wants? 

(Yes)  (8.a)  How  are  you  resolving  this? 

[9]  Do  you  and  the  customer  understand  the  same 
thing  by  the  requirements? 

(Yes)  (9.a)  Is  there  a  process  by  which  to  deter¬ 
mine  this? 

[1 0]  How  do  you  validate  the  requirements? 

•  Prototyping 

•  Analysis 

•  Simulations 

e)  Feasibility 

[Are  requirements  infeasible  from  an  analytical  point  of 
view?] 

[1 1  ]  Are  there  any  requirements  that  are  technically 
difficult  to  implement? 

(Yes)  (11. a)  What  are  they? 

(Yes)  (1 1.b)Why  are  they  difficult  to  implement? 

(No)  (1 1  .c)  Were  feasibility  studies  done  for 
these  requirements? 
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(Yes)  (1 1  .c.1 )  How  confident  are  you  of 
the  assumptions  made  in 
the  studies? 


f)  Precedent 

[Do  requirements  specify  something  never  done  before,  or 
that  your  company  has  not  done  before?] 

[1 2]  Are  there  any  state-of-the-art  requirements? 

•  Technologies 

•  Methods 

•  Languages 

•  Hardware 

(No)  (1 2.a)  Are  any  of  these  new  to  you? 

(Yes)  (12.b)  Does  the  program  have  sufficient 
knowledge  in  these  areas? 

(No)  (12.b.1)  Is  there  a  plan  for  acquir¬ 
ing  knowledge  in  these 
areas? 

g)  Scale 

[Do  requirements  specify  a  product  larger,  more  complex, 
or  requiring  a  larger  organization  than  in  the  experience  of 
the  company?] 

[1 3]  Is  the  system  size  and  complexity  a  concern? 

(No)  (1 3.a)  Have  you  done  something  of  this 
size  and  complexity  before? 

[1 4]  Does  the  size  require  a  larger  organization  than 
usual  for  your  company? 

2.  Design 

a)  Functionality 

[Are  there  any  potential  problems  in  meeting  functionality 
requirements?] 

[1 5]  Are  there  any  specified  algorithms  which  may 
not  satisfy  the  requirements? 
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(No)  (1 5.a)  Are  any  of  the  algorithms  or  design 
marginal  with  respect  to  meeting 
requirements? 

[1 6]  How  do  you  determine  the  feasibility  of  algo¬ 
rithms  and  designs? 

•  Prototyping 

•  Modeling 

•  Analysis 

•  Simulation 

b)  Difficulty 

[Will  the  design  and/or  implementation  be  difficult  to 
achieve?] 

[1 7]  Does  any  of  the  design  depend  on  unrealistic  or 
optimistic  assumptions? 

[1 8]  Are  there  any  requirements  or  functions  which 
are  difficult  to  design? 

(No)  (1 8.a)  Do  you  have  solutions  for  all  the  re¬ 
quirements? 

(Yes)  (18.b)  What  are  the  requirements? 

•  Why  are  they  difficult? 

c)  Interfaces 

[Are  the  internal  interfaces  (hardware  and  software)  well 
defined  and  controlled?] 

[1 9]  Are  the  internal  interfaces  well  defined? 

•  Software-to-software 

•  Software-to-hardware 

[20]  Is  there  a  process  for  defining  internal  inter¬ 
faces? 

(Yes)  (20. a)  Is  there  a  change  control  process 
for  internal  interfaces? 

[21]  Is  hardware  being  developed  in  parallel  with  soft¬ 
ware? 
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(Yes)  (21  .a)  Are  the  hardware  specifications 
changing? 

(Yes)  (21  .b)  Have  all  the  interfaces  to  software 
been  defined? 

(Yes)  (21  .c)  Will  there  be  engineering  design 
models  that  can  be  used  to  test 
the  software? 

d)  Performance 

[Are  there  stringent  response  time  or  throughput  require¬ 
ments?] 

[22]  Are  there  any  problems  with  performance? 

•  Throughput 

•  Scheduling  asynchronous 
real-time  events 

•  Real-time  response 

•  Recovery  timelines 

•  Response  time 

•  Database  response,  con¬ 
tention,  or  access 

[23]  Has  a  performance  analysis  been  done? 

(Yes)  (23.a)  What  is  your  level  of  confidence  in 
the  performance  analysis? 

(Yes)  (23. b)  Do  you  have  a  model  to  track  per 
formance  through  design  and 
implementation? 

e)  Testability 

[Is  the  product  difficult  or  impossible  to  test?] 

[24]  Is  the  software  going  to  be  easy  to  test? 

[25]  Does  the  design  include  features  to  aid  testing? 

[26]  Do  the  testers  get  involved  in  analyzing  require¬ 
ments? 
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f)  Hardware  Constraints 

[Are  there  tight  constraints  on  the  target  hardware?] 

[27]  Does  the  hardware  limit  your  ability  to  meet  any 
requirements? 

•  Architecture 

•  Memory  capacity 
•Throughput 

•  Real-time  response 

•  Response  time 

•  Recovery  timelines 

•  Database  performance 

•  Functionality 

•  Reliability 

•  Availability 

g)  Non-Developmental-Software 

[Are  there  problems  with  software  used  in  the  program  but 
not  developed  by  the  program?] 

If  re-used  or  re-engineered  software 
exists 

[28]  Are  you  reusing  or  reengineering  software  not 
developed  on  the  program? 

(Yes)  (28.a)  Do  you  foresee  any  problems? 

•  Documentation 

•  Performance 

•  Functionality 
•Timely  delivery 

•  Customization 
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If  COTS  software  is  being  used 

[29]  Are  there  any  problems  with  using  COTS  (com¬ 
mercial  off-the-shelf)  software? 

•  Insufficient  documentation 
to  determine  interfaces, 
size,  or  performance 

•  Poor  performance 

•  Requires  a  large  share  of 
memory  or  database  stor¬ 
age. 

•  Difficult  to  interface  with 
application  software 

•  Not  thoroughly  tested 

•  Not  bug  free 

•  Not  maintained  adequately 

•  Slow  vendor  response 

[30]  Do  you  foresee  any  problem  with  integrating 
COTS  software  updates  or  revisions? 

3.  Code  and  Unit  Test 

a)  Feasibility 

[Is  the  implementation  of  the  design  difficult  or  impossi¬ 
ble?] 

[31  ]  Are  any  parts  of  the  product  implementation  not 
completely  defined  by  the  design  specification? 

[32]  Are  the  selected  algorithms  and  designs  easy  to 
implement? 

b)  Testing 

[Is  the  specified  level  and  time  for  unit  testing  adequate?] 

[33]  Do  you  begin  unit  testing  before  you  verify  code 
with  respect  to  the  design 

[34]  Has  sufficient  unit  testing  been  specified? 
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[35]  Is  there  sufficient  time  to  perform  all  the  unit  test¬ 
ing  you  think  should  be  done? 

[36]  Will  compromises  be  made  regarding  unit  testing 
if  there  are  schedule  problems? 

c)  Coding/Implementation 

[Are  there  any  problems  with  coding  and  implementation?] 

[37]  Are  the  design  specifications  in  sufficient  detail 
to  write  the  code? 

[38]  Is  the  design  changing  while  coding  is  beinq 
done? 

[39]  Are  there  system  constraints  making  the  code 
difficult  to  write? 

•Timing 

•  Memory 

•  External  storage 

[40]  Is  the  language  suitable  for  producing  the  soft¬ 
ware  on  this  program? 

[41]  Are  there  multiple  languages  used  on  the  pro¬ 
gram? 

(Yes)  (41  .a)  Is  there  interface  compatibility  be¬ 
tween  the  code  produced  by  the 
different  compilers? 

[42]  Is  the  development  computer  the  same  as  the 
target  computer? 

(No)  (42. a)  Are  there  compiler  differences  be¬ 
tween  the  two? 
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If  developmental  hardware  is  being 
used 

[43]  Are  the  hardware  specifications  adequate  to 
code  the  software? 

[44]  Are  the  hardware  specifications  changing  while 
the  code  is  being  written? 

4.  Integration  and  Test 

a)  Environment 

[Is  the  integration  and  test  environment  adequate?] 

[45]  Will  there  be  sufficient  hardware  to  do  adequate 
integration  and  testing? 

[46]  Is  there  any  problem  with  developing  realistic 
scenarios  and  test  data  to  demonstrate  any  re¬ 
quirements? 

•  Specified  data  traffic 

•  Real-time  response 

•  Asynchronous  event  han¬ 
dling 

•  Multi-user  interaction 

[47]  Are  you  able  to  verify  performance  in  your  facili¬ 
ty? 

[48]  Does  hardware  and  software  instrumentation  fa¬ 
cilitate  testing? 

(Yes)  (48.a)ls  it  sufficient  for  all  testing? 

b)  Product 

[Is  the  interface  definition  inadequate,  facilities  inade¬ 
quate,  time  insufficient?] 

[49]  Will  the  target  hardware  be  available  when  need¬ 
ed? 
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[50]  Have  acceptance  criteria  been  agreed  to  for  all 
requirements? 

(Yes)  (50.a)  Is  there  a  formal  agreement? 

[51  ]  Are  the  external  interfaces  defined,  documented, 
and  baselined? 

[52]  Are  there  any  requirements  that  will  be  difficult  to 
test? 

[53]  Has  sufficient  product  integration  been  speci¬ 
fied? 

[54]  Has  adequate  time  been  allocated  for  product  in¬ 
tegration  and  test? 

If  COTS 

[55]  Will  vendor  data  be  accepted  in  verification  of  re¬ 
quirements  allocated  to  COTS  products? 

(Yes)  (55. a)  Is  the  contract  clear  on  that? 
c)  System 

[System  integration  uncoordinated,  poor  interface  defini¬ 
tion,  or  inadequate  facilities?] 

[56]  Has  sufficient  system  integration  been  speci¬ 
fied? 

[57]  Has  adequate  time  been  allocated  for  system  in¬ 
tegration  and  test? 

[58]  Are  all  contractors  part  of  the  integration  team? 

[59]  Will  the  product  be  integrated  into  an  existing 
system? 

(Yes)  (59. a)  Is  there  a  parallel  cutover  period 
with  the  existing  system? 

(No)  (59. a.  1)  How  will  you  guarantee 
the  product  will  work  cor¬ 
rectly  when  integrated? 
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[60]  Will  system  integration  occur  on  customer  site? 

5.  Engineering  Specialties 

a)  Maintainability 

[Will  the  implementation  difficult  to  understand  or  main¬ 
tain?] 

[61]  Does  the  architecture,  design,  or  code  create 
any  maintenance  difficulties? 

[62]  Are  the  maintenance  people  involved  early  in  the 
design? 

[63]  Is  the  product  documentation  adequate  for  main¬ 
tenance  by  an  outside  organization? 

b)  Reliability 

[Are  the  reliability  or  availability  requirements  difficult  to 
meet?] 

[64]  Are  reliability  requirements  allocated  to  the  soft¬ 
ware? 

[65]  Are  availability  requirements  allocated  to  the 
software? 

(Yes)  (65.a)  Are  recovery  timelines  any  prob¬ 
lem? 

c)  Safety 

[Are  the  safety  requirements  infeasible  and  not  demon¬ 
strable?] 

[66]  Are  safety  requirements  allocated  to  the  soft¬ 
ware? 

(Yes)  (66. a)  Do  you  see  any  difficulty  in  meeting 
the  safety  requirements? 

[67]  Will  it  be  difficult  to  verify  satisfaction  of  safety  re¬ 
quirements? 
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d)  Security 

[Are  the  security  requirements  more  stringent  than  the 
current  state  of  the  practice  or  program  experience?] 

[68]  Are  there  unprecedented  or  state-of-the-art  se¬ 
curity  requirements? 

[69]  Is  it  an  Orange  Book  system? 

[70]  Have  you  implemented  this  level  of  security  be¬ 
fore? 

e)  Human  Factors 

[Will  the  system  will  be  difficult  to  use  because  of  poor  hu¬ 
man  interface  definition?] 

[71]  Do  you  see  any  difficulty  in  meeting  the  Human 
Factors  requirements? 

(No)  (71  .a)  How  are  you  ensuring  that  you  will 
meet  the  human  interface  require¬ 
ments? 


If  prototyping 

(Yes)  (71  .a.1 )  Is  it  a  throw-away  proto¬ 
type? 

(No)  (71  a.2)  Are  you  doing  evolu¬ 
tionary  develop¬ 
ment? 

(Yes)(71.b)  Are  you  experi¬ 
enced  in  this 
type  of  develop¬ 
ment? 

(Yes)(71.c)  Are  interim  ver¬ 
sions  deliver¬ 
able? 

(Yes)(71  .d)  Does  this  com¬ 
plicate  change 
control? 
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f)  Specifications 

[Is  the  documentation  adequate  to  design,  implement,  and 
test  the  system?] 

[72]  Is  the  software  requirements  specification  ade¬ 
quate  to  design  the  system? 

[73]  Are  the  hardware  specifications  adequate  to  de¬ 
sign  and  implement  the  software? 

[74]  Are  the  external  interface  requirements  well 
specified? 

[75]  Are  the  test  specifications  adequate  to  fully  test 
the  system? 

If  in  or  past  implementation  phase 

[76]  Are  the  design  specifications  adequate  to  imple¬ 
ment  the  system? 

•  Internal  interfaces 
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B.  Development  Environment 


mmmmmmssmm 


6.  Development  Process 

a)  Formality 

[Will  the  implementation  difficult  to  understand  or  main¬ 
tain?] 

[77]  Is  there  more  than  one  development  model  be¬ 
ing  used? 

•  Spiral 

•  Waterfall 

•  Incremental 

(Yes)  (77.a)  Is  coordination  between  them  a 
problem? 

[78]  Are  there  formal,  controlled  plans  for  all  develop¬ 
ment  activities? 

•  Requirements  analysis 

•  Design 

•  Code 

•  Integration  and  test 

•  Installation 

•  Quality  assurance 

•  Configuration  management 
(Yes)  (78. a)  Do  the  plans  specify  the  process 

well? 

(Yes)  (78.b)  Are  developers  familiar  with  the 
plans? 

b)  Suitability 

[Is  the  process  suited  to  the  development  model,  e.g.,  spi¬ 
ral,  prototyping?] 

[79]  Is  the  development  process  adequate  for  this 
product? 
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[80]  Is  the  development  process  supported  by  a  com¬ 
patible  set  of  procedures,  methods  and  tools? 

c)  Process  Control 

[Is  the  software  development  process  enforced,  monitored 
and  controlled  using  metrics?  Are  distributed  development 
sites  coordinated?] 

[81  ]  Does  everyone  follow  the  development  process? 
(Yes)  (81  .a)  How  is  this  insured? 

[82]  Can  you  measure  whether  the  development  pro¬ 
cess  is  meeting  your  productivity  and  quality 
goals? 

If  there  are  distributed  development 
sites 

[83]  Is  there  adequate  coordination  among  distribut¬ 
ed  development  sites? 

d)  Familiarity 

[Are  the  project  members  experienced  in  use  of  the  pro¬ 
cess?  Is  the  process  understood  by  all  staff  members?] 

[84]  Are  people  comfortable  with  the  development 
process? 

e)  Product  Control 

[Are  there  mechanisms  for  controlling  changes  in  the 
product?] 

[85]  Is  there  a  requirements  traceability  mechanism 
that  tracks  requirements  from  the  source  specifi¬ 
cation  through  test  cases? 

[86]  Is  the  traceability  mechanism  used  in  evaluating 
requirement  change  impact  analyses? 
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[87]  Is  there  a  formal  change  control  process? 

(Yes)  (87.a)  Does  it  cover  all  changes  to  base¬ 
lined  requirements,  design,  code, 
and  documentation? 

[88]  Are  changes  at  any  level  mapped  up  to  the  sys¬ 
tem  level  and  down  through  the  test  level? 

[89]  Is  there  adequate  analysis  when  new  require¬ 
ments  are  added  to  the  system? 

[90]  Do  you  have  a  way  to  track  interfaces? 

[91  ]  Are  the  test  plans  and  procedures  updated  as 
part  of  the  change  process? 

7.  Development  System 

a)  Capacity 

[Is  there  sufficient  work  station  processing  power,  memo¬ 
ry,  or  storage  capacity?] 

[92]  Are  there  enough  workstations  and  processing 
capacity  for  all  staff? 

[93]  Is  there  sufficient  capacity  for  overlapping  phas¬ 
es,  such  as  coding,  integration  and  test? 

b)  Suitability 

[Does  the  development  system  support  all  phases,  activi¬ 
ties,  and  functions?] 

[94]  Does  the  development  system  support  all  as¬ 
pects  of  the  program? 

•  Requirements  analysis 

•  Performance  analysis 

•  Design 

•  Coding 
•Test 

•  Documentation 
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•  Configuration  management 

•  Management  tracking 

•  Requirements  traceability 

c)  Usability 

[How  easy  is  the  development  system  to  use?] 

[95]  Do  people  find  the  development  system  easy  to 
use? 

[96]  Is  there  good  documentation  of  the  development 
system? 

d)  Familiarity 

[Little  prior  company  or  project  member  experience  with 
the  development  system?] 

[97]  Have  people  used  these  tools  and  methods  be¬ 
fore? 

e)  Reliability 

[Does  the  system  suffer  from  software  bugs,  down-time, 
insufficient  built-in  back-up?] 

[98]  Is  the  system  considered  reliable? 

•  Compiler 

•  Development  tools 

•  Hardware 

f)  System  Support 

[Is  there  timely  expert  or  vendor  support  for  system?] 

[99]  Are  the  people  trained  in  use  of  the  development 
tools? 

[1 00]  Do  you  have  access  to  experts  in  use  of  the  sys¬ 
tem? 

[101]  Do  the  vendors  respond  to  problems  rapidly? 
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g)  Deliverability 

[Are  the  definition  and  acceptance  requirements  defined 
for  delivering  the  development  system  to  the  customer;  not 
budgeted?  HINT:  if  the  participants  are  confused  about 
this,  it  is  probably  not  an  issue  from  a  risk  perspective.] 

[102]  Are  you  delivering  the  development  system  to 
the  customer? 

(Yes)  (1 02.a)  Have  adequate  budget,  schedule, 
and  resources  been  allocated  for 
this  deliverable? 

8.  Management  Process 

a)  Planning 

[Is  the  planning  timely  technical  leads  included  contingen¬ 
cy  planning  done?] 

[103]  Is  the  program  managed  according  to  the  plan? 
(Yes)  (103.a)  Do  people  routinely  get  pulled 

away  to  fight  fires? 

[104]  Is  re-planning  done  when  disruptions  occur? 

[105]  Are  people  at  all  levels  included  in  the  planning 
of  their  own  work? 

[106]  Are  there  contingency  plans  for  known  risks? 
(Yes)  (1 06.a)  How  do  you  determine  when  to 

activate  the  contingencies? 

[107]  Are  long-term  issues  being  adequately  ad¬ 
dressed? 

b)  Project  Organization 

[Are  the  roles  and  reporting  relationships  clear?] 

[108]  Is  the  program  organization  effective? 

[1 09]  Do  people  understand  their  own  and  others  roles 
in  the  program? 
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[1 1 0]  Do  people  know  who  has  authority  for  what? 

c)  Management  Experience 

[Are  the  managers  experienced  in  software  development, 
software  management,  the  application  domain,  the  devel¬ 
opment  process,  or  on  large  programs?] 

[111]  Does  the  program  have  experienced  managers? 

•  Software  management 

•  Hands-on  software  devel¬ 
opment 

•  With  this  development  pro¬ 
cess 

•  In  the  application  domain 

•  Program  size  or  complexity 

d)  Program  Interfaces 

[Poor  interface  with  customer,  other  contractors,  senior 
and/or  peer  managers.] 

[112]  Does  management  communicate  problems  up 
and  down  the  line? 

[1 1 3]  Are  conflicts  with  the  customer  documented  and 
resolved  in  a  timely  manner? 

[1 1 4]  Does  management  involve  appropriate  program 
members  in  meetings  with  the  customer? 

•Technical  leaders 

•  Developers 

•  Analysts 

[1 1 5]  Does  management  work  to  ensure  that  all  cus¬ 
tomer  factions  are  represented  in  decisions  re¬ 
garding  functionality  and  operation? 

[116]  Does  the  project  always  present  an  optimistic 
picture  to  the  customer  or  senior  management? 
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9.  Management  Methods 

a)  Monitoring 

[Are  management  metrics  defined;  development  progress 
tracked?] 

[117]  Are  there  periodic  structured  status  reports? 
(Yes)  (1 1 7.a)  Do  people  get  a  response  to  their 

status  reports? 

[1 1 8]  Does  appropriate  information  get  reported  to  the 
right  organizational  levels? 

[1 1 9]  Do  you  track  progress  versus  plan? 

(Yes)  (1 1 9.a)  Does  management  have  a  clear 
picture  of  what  is  going  on? 

b)  Personnel  Management 

[Are  project  personnel  trained  and  used  appropriately?] 

[1 20]  Do  people  get  trained  in  skills  required  for  this 
program? 

(Yes)  (1 20.a)  Is  this  part  of  the  program  plan? 

[121]  Do  people  get  assigned  to  the  program  who  do 
not  match  the  experience  profile  for  your  work 
area? 

[1 22]  Is  it  easy  for  program  members  to  get  manage¬ 
ment  action? 

[1 23]  Are  program  members  at  all  levels  aware  of  their 
status  versus  plan? 

[124]  Do  people  assiduously  keep  to  the  plan? 

[125]  Does  management  consult  with  people  before 
making  decisions  that  affect  their  work? 
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[126]  Does  program  management  involve  appropriate 
program  members  in  meetings  with  the  custom¬ 
er? 

•Technical  leaders 

•  Developers 

•  Analysts 

c)  Quality  Assurance 

[Are  there  adequate  procedures  and  resources  to  assure 
product  quality?] 

[1 27]  Is  the  Software  Quality  Assurance  function  ade¬ 
quately  staffed  on  this  program? 

[128]  Do  you  have  defined  mechanisms  for  assuring 
quality? 

(Yes)  (1 28.a)  Do  all  areas  and  phases  have 
quality  procedures? 

(Yes)  (1 28.b)  Are  people  used  to  working  with 
these  procedures? 

d)  Configuration  Management 

[Are  the  change  procedures  or  version  control,  including 
installation  site(s)  adequate?] 

[1 29]  Do  you  have  an  adequate  configuration  manage¬ 
ment  system? 

[130]  Is  the  Configuration  Management  function  ade¬ 
quately  staffed? 

[131]  Is  coordination  required  with  an  installed  sys¬ 
tem? 

(Yes)  (1 31  .a)  Is  there  adequate  configuration 
management  of  the  installed  sys¬ 
tem? 

(Yes)  (1 31  .b)  Does  the  configuration  manage¬ 
ment  system  synchronize  your 
work  with  site  changes? 
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[132]  Are  you  installing  in  multiple  sites? 

(Yes)  (1 32.a)  Does  the  configuration  manage 
ment  system  provide  for  multiple 
sites? 

5.  Work  Environment 

a)  Quality  Attitude 

[Is  there  a  lack  of  orientation  toward  quality  workman¬ 
ship?] 

[1 33]  Are  all  staff  levels  oriented  toward  quality  proce¬ 
dures? 

[1 34]  Does  schedule  get  in  the  way  of  quality? 

b)  Cooperation 

[Is  there  a  lack  of  team  spirit;  does  conflict  resolution  re¬ 
quire  management  intervention?] 

[1 35]  Do  people  work  cooperatively  across  functional 
boundaries? 

[1 36]  Do  people  work  effectively  towards  common 
goals? 

[1 37]  Is  management  intervention  sometimes  required 
to  get  people  working  together? 

c)  Communication 

[Is  there  poor  awareness  of  mission  or  goals;  poor  com¬ 
munication  of  technical  information  among  peers  and 
managers?] 

[138]  Is  there  good  communication  among  the  mem¬ 
bers  of  the  program? 

•  Managers 

•  Technical  leaders 

•  Developers 
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•Testers 

•  Configuration  management 

•  Quality  assurance 

[139]  Are  the  managers  receptive  to  communication 
from  program  staff? 

(Yes)  (1 39. a)  Do  you  feel  free  to  ask  your  man¬ 
agers  for  help? 

(Yes)  (1 39. b)  Are  members  of  the  program  able 
to  raise  risks  without  having  a 
solution  in  hand? 

[140]  Do  the  program  members  get  timely  notification 
of  events  which  may  affect  their  work? 

(Yes)  (1 40.a)  Is  this  formal  or  informal? 

d)  Morale 

[Is  there  a  non-productive,  non-creative  atmosphere?  Do 
people  feel  that  there  is  no  recognition  or  reward  for  supe¬ 
rior  work?] 

[141]  How  is  morale  on  the  program? 

(No)  (1 41  .a)  What  is  the  main  contributing  fac¬ 
tor  to  low  morale? 

[142]  Is  there  any  problem  keeping  the  people  you 
need? 


B-26 


CMU/SEI-94-TR-19 


Appendix  B 


C.  Program  Constraints 


1.  Resources 

a)  Schedule 

[Is  the  schedule  inadequate  or  unstable?] 

[1 43]  Has  the  schedule  been  stable? 

[144]  Is  the  schedule  realistic? 

(Yes)  (144.a)  Is  the  estimation  method  based 
on  historical  data? 

(Yes)  (1 44.b)  Has  the  method  worked  well  in  the 
past? 

[1 45]  Is  there  anything  for  which  adequate  schedule 
was  not  planned? 

•  Analysis  and  studies 
•QA 

•  Training 

•  Maintenance  courses  and 
training 

•  Capital  equipment 

•  Deliverable  development 
system 

[1 46]  Are  there  external  dependencies  which  are  likely 
to  impact  the  schedule? 

b)  Staff 

[Is  the  staff  inexperienced,  lacking  domain  knowledge, 
lacking  skills,  or  understaffed?] 

[1 47]  Are  there  any  areas  where  the  required  technical 
skills  are  lacking? 

•  Software  engineering  and 
requirements  analysis 
method 
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•  Algorithm  expertise 

•  Design  and  design  meth¬ 
ods 

•  Programming  languages 

•  Integration  and  test  meth¬ 
ods 

•  Reliability 

•  Maintainability 

•  Availability 

•  Human  factors 

•  Configuration  management 

•  Quality  assurance 

•  Target  environment 

•  Level  of  security 
•COTS 

•  Reuse  software 

•  Operating  system 

•  Database 

•  Application  domain 

•  Performance  analysis 

•  Time-critical  applications 

[1 48]  Do  you  have  adequate  personnel  to  staff  the  pro¬ 
gram? 

[149]  Is  the  staffing  stable? 

[1 50]  Do  you  have  access  to  the  right  people  when  you 
need  them? 

[1 51  ]  Have  the  program  members  implemented  similar 
systems  of  this  type? 

[152]  Is  the  program  reliant  on  a  few  key  people? 

[153]  Is  there  any  problem  with  getting  cleared  peo¬ 
ple? 
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c)  Budget 

[Is  the  funding  insufficient  or  unstable?] 

[1 54]  Is  the  budget  stable? 

[155]  Is  the  budget  based  on  a  realistic  estimate? 
(Yes)  (1 55.a)  Is  the  estimation  method  based 

on  historical  data? 

(Yes)  (1 55.b)  Has  the  method  worked  well  in  the 
past? 

[1 56]  Have  features  or  functions  been  deleted  as  part 
of  a  design-to-cost  effort? 

[1 57]  Is  there  anything  for  which  adequate  budget  was 
not  allocated? 

•  Analysis  and  studies 
•QA 

•  Training 

•  Maintenance  courses 

•  Capital  equipment 

•  Deliverable  development 
system 

[158]  Do  budget  changes  accompany  requirement 
changes? 

(Yes)  (1 58. a)  Is  this  a  standard  part  of  the 
change  control  process? 


d)  Facilities 

[Are  the  facilities  adequate  for  building  and  delivering  the 
product?] 

[159]  Are  the  development  facilities  adequate? 

[160]  Is  the  integration  environment  adequate? 
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2.  Contract 

a)  Type  of  Contract 

{Is  the  contract  type  a  source  of  risk  to  the  program?] 

[1 61]  What  type  of  contract  do  you  have?  (Cost  plus 
award  fee,  fixed  price,....) 

(161a)  Does  this  present  any  problems? 

[1 62]  Is  the  contract  burdensome  in  any  aspects  of  the 
program? 

•  SOW  (Statement  of  Work) 

•  Specifications 

•  Data  Item  Descriptions 

•  Contract  parts 

•  Excessive  customer  in¬ 
volvement 

[1 63]  Is  the  required  documentation  burdensome? 

•  Excessive  amount 

•  Picky  customer 

•  Long  approval  cycle 

b)  Restrictions 

[Does  the  contract  cause  any  restrictions?] 

[164]  Are  there  problems  with  data  rights? 

•  COTS  software 

•  Developmental  software 

•  Non-developmental  Items 

c)  Dependencies 

[Does  the  program  have  any  dependencies  on  outside 
products  or  services?] 

[165]  Are  there  dependencies  on  external  products  or 
services  which  may  affect  the  product,  budget  or 
schedule? 

•  Associate  contractors 
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•  Prime  contractor 

•  Subcontractors 

•  Vendors  or  suppliers 

•  Customer  furnished  equip¬ 
ment  or  software 

3.  Program  Interfaces 

a)  Customer 

[Are  there  any  customer  problems  such  as:  lengthy  docu¬ 
ment-approval  cycle,  poor  communication,  and  inade¬ 
quate  domain  expertise?] 

[1 66]  Is  the  customer  approval  cycle  timely? 

•  Documentation 

•  Program  reviews 

•  Formal  reviews 

[1 67]  Do  you  ever  proceed  before  receiving  customer 
approval? 

[1 68]  Does  the  customer  understand  the  technical  as¬ 
pects  of  the  system? 

[1 69]  Does  the  customer  understand  software? 

[1 70]  Does  the  customer  interfere  with  process  or  peo¬ 
ple? 

[1 71  ]  Does  management  work  with  the  customer  to 
reach  mutually  agreeable  decisions  in  a  timely 
manner? 

•  Requirements  understand¬ 
ing 

•  Test  criteria 

•  Schedule  adjustments 

•  Interfaces 
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[1 72]  How  effective  are  your  mechanisms  for  reaching 
agreements  with  the  customer? 

•  Working  groups  (Contrac¬ 
tual?) 

•Technical  Interchange 
Meetings  (Contractual?) 

[173]  Are  all  customer  factions  involved  in  reaching 
agreements? 

(Yes)  (1 73. a)  Is  it  a  formally  defined  process? 

[1 74]  Does  management  present  a  realistic  or  optimis¬ 
tic  picture  to  the  customer? 

If  there  are  associate  contractors 

b)  Associate  Contractors 

[Are  there  any  problems  with  associate  contractors  such 
as  inadequately  defined  or  unstable  interfaces,  poor  com¬ 
munication  or  cooperation?] 

[175]  Are  the  external  interfaces  changing  without  ad¬ 
equate  notification,  coordination  or  formal 
change  procedures? 

[176]  Is  there  an  adequate  transition  plan? 

(Yes)  (176.a)  Is  it  supported  by  all  contractors 
and  site  personnel? 

[1 77]  Is  there  any  problem  with  getting  schedules  or  in¬ 
terface  data  from  associate  contractors? 

(No)  (1 77. a)  Are  they  accurate? 

If  there  are  sub-contractors 

c)  Subcontractors 

[Is  the  program  dependent  on  subcontractors  for  any  crit¬ 
ical  areas?] 

[178]  Are  there  any  ambiguities  in  subcontractor  task 
definitions? 
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[179]  Is  the  subcontractor  reporting  and  monitoring 
procedure  different  from  the  program’s  reporting 
requirements? 

[1 80]  Is  subcontractor  administration  and  technical 
management  done  by  a  separate  organization? 

[181]  Are  you  highly  dependent  on  subcontractor  ex¬ 
pertise  in  any.  areas? 

[1 82]  Is  subcontractor  knowledge  being  transferred  to 
the  company? 

[1 83]  Is  there  any  problem  with  getting  schedules  or  in¬ 
terface  data  from  subcontractors? 

If  program  is  a  sub-contract 

d)  Prime  Contractor 

[Is  the  program  facing  difficulties  with  its  Prime  contrac¬ 
tor?] 

[1 84]  Are  your  task  definitions  from  the  Prime  ambigu¬ 
ous? 

[1 85]  Do  you  interface  with  two  separate  prime  organi¬ 
zations  for  administration  and  technical  manage¬ 
ment? 

[1 86]  Are  you  highly  dependent  on  Prime’s  expertise  in 
any  areas? 

[1 87]  Is  there  any  problem  with  getting  schedules  or  in¬ 
terface  data  from  the  Prime? 

e)  Corporate  Management 

[Is  there  a  lack  of  support  or  micro  management  from  up¬ 
per  management?] 

[188]  Does  program  management  communicate  prob¬ 
lems  to  senior  management? 

(Yes)  (1 88.a)  Does  this  seem  to  be  effective? 
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[189]  Does  corporate  management  give  you  timely 
support  in  solving  your  problems? 

[1 90]  Does  corporate  management  tend  to  micro-man- 
age? 

[191]  Does  management  present  a  realistic  or  optimis¬ 
tic  picture  to  senior  management? 

f)  Vendors 

[Are  vendors  responsive  to  programs  needs?] 

[192]  Are  you  relying  on  vendors  for  deliveries  of  criti¬ 
cal  components? 

•  Compilers 

•  Hardware 
•COTS 

g)  Politics 

[Are  politics  causing  a  problem  for  the  program?] 

[193]  Are  politics  affecting  the  program? 

•  Company 

•  Customer 

•  Associate  contractors 

•  Subcontractors 

[194]  Are  politics  affecting  technical  decisions? 
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Application  domain  -  A  term  that  refers  to  the  nature 
of  the  application.  Examples  are  real-time  flight  control 
systems  and  management  information  systems. 

Availability-  The  relative  time  that  an  operational 
product  must  be  available  for  use.  Usually  expressed 
as  the  ratio  of  time  available  for  use  to  some  total  time 
period  or  as  specific  hours  of  operation. 

Change  control -A  part  of  configuration  management 
that  reviews,  approves,  and  tracks  progress  of  alter¬ 
ations  in  the  configuration  of  a  configuration  item  deliv¬ 
ered,  to  be  delivered,  or  under  formal  development,  af¬ 
ter  formally  establishing  its  configuration  identification. 

Configuration  management-  A  discipline  applying 
technical  and  administrative  direction  and  surveillance 
to  identify  and  document  the  functional  and  physical 
characteristics  of  a  controlled  item,  control  changes  to 
a  configuration  item  and  its  documentation,  and  record 
and  report  change  processing  and  implementation  sta¬ 
tus. 

Configuration  management  system  -  The  process¬ 
es,  procedures,  and  tools  used  by  the  development  or¬ 
ganization  to  accomplish  configuration  management. 

COTS  (commercial  off-the-shelf)  -  A  type  of  non-de- 
velopmental  software  that  is  supplied  by  commercial 
sources. 
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Customer  -  The  person  or  organization  receiving  a 
product  or  service.  There  may  be  many  different  cus¬ 
tomers  for  individual  organizations  within  a  program 
structure.  Government  program  offices  may  view  the 
customer  as  the  user  organization  for  which  they  are 
managing  the  project.  Contractors  may  view  the  pro¬ 
gram  office  as  well  as  the  user  organization  as  custom¬ 
ers. 

Design  specifications  -  A  document  that  prescribes 
the  form,  parts,  and  details  of  the  product  according  to 
a  plan. 

Design-to-cost-  A  term  describing  the  bidding  of  a 
selected,  reduced  set  of  requirements  to  meet  cost  ob¬ 
jectives. 

Development  computer-  The  hardware  and  support¬ 
ing  software  system  used  for  software  development. 

Development  facilities  -  The  office  space,  furnish¬ 
ings,  and  equipment  that  support  the  development 
staff. 

Development  model—  The  abstract  visualization  of 
how  the  software  development  functions  (such  as  re¬ 
quirements  definition,  design,  code,  test,  and  imple¬ 
mentation)  are  organized.  Typical  models  are  the  wa¬ 
terfall  model,  the  iterative  model,  and  the  spiral  model. 
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Development  process  -  The  implemented  process 
for  managing  the  development  of  the  deliverable  prod¬ 
uct.  For  software,  the  development  process  includes 
the  following  major  activities,  which  may  overlap  and 
may  be  applied  iteratively  or  recursively: 

•  System  Requirements  Analysis/Design 

•  Software  Requirements  Analysis 

•  Preliminary  Design 

•  Detailed  Design 

•  Coding  and  Computer  Software  Unit  Testing 

•  Computer  Software  Component  Integration 
and  Testing 

•  Computer  Software  Configuration  Item  Test¬ 
ing 

•  System  Integration  and  Testing 

Development  sites-  The  locations  at  which  develop¬ 
ment  work  is  being  conducted. 

Development  system -The  hardware  and  software 
tools  and  supporting  equipment  that  will  be  used  in 
product  development  including  such  items  as  computer 
aided  software  engineering  (CASE)  tools,  compilers, 
and  configuration  management  systems. 

External  dependencies  -  Any  deliverable  product  or 
service  from  other  organizations  that  is  critical  to  the 
project  or  program. 

External  interfaces  -  The  points  where  the  software 
system  under  development  interacts  with  other  sys¬ 
tems,  sites,  or  people. 

Hardware  specifications-  A  document  that  pre¬ 
scribes  the  functions,  materials,  dimensions,  and  work¬ 
manship  that  a  hardware  item  must  meet. 
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Implementation  -  The  act  of  preparing  the  product  for 
use  by  the  customer. 

Integration  -  The  act  of  assembling  individual  hard¬ 
ware  and/or  software  components  into  a  usable  whole. 

Integration  environment-  The  hardware,  software, 
and  supporting  tools  that  will  be  used  to  support  prod¬ 
uct  integration. 

Internal  interfaces  -  The  points  where  the  software 
system  underdevelopment  interacts  with  other  compo¬ 
nents  of  the  system  under  development. 

Long-term  issues  -  Issues  of  strategic  importance  to 
the  project  that  can  be  compromised  in  the  heat  of  bat¬ 
tle.  Issues  such  as  employee  training  and  develop¬ 
ment,  establishing  and  improving  processes  and  pro¬ 
cedures,  and  similar  activities  are  important  to  the  long¬ 
term  viability  of  the  project  and  the  organization. 

Non-developmental  software  (NDS)  -  Deliverable 
software  that  is  not  developed  under  the  contract  but  is 
provided  by  the  contractor,  the  government,  or  a  third 
party.  NDS  may  be  referred  to  as  reusable  software, 
government-furnished  software,  or  commercially  avail¬ 
able  software,  depending  on  its  source. 

Orange  Book -A  security  standard  set  by  the  U.S. 
government. 

Product  integration  -  The  act  of  assembling  individu¬ 
al  hardware  and  software  components  into  a  functional 
whole. 
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Re-engineering  -  The  practice  of  adapting  existing 
software  artifacts  or  systems  to  perform  new  or  en¬ 
hanced  functions.  Re-engineered  artifacts  may  be  sub¬ 
stantially  different  from  existing  artifacts. 

Reliability  -The  degree  of  dependability  that  an  oper¬ 
ational  product  must  meet.  Usually  expressed  as  the 
average  time  to  failure. 

Reusing  -  Hardware  or  software  developed  in  re¬ 
sponse  to  the  requirements  of  one  application  that  can 
be  used,  in  whole  or  in  part,  to  satisfy  the  requirements 
of  another  application. 

Safety- The  degree  to  which  the  software  product 
minimizes  the  potential  for  hazardous  conditions  during 
its  operational  mission. 

Security-  The  degree  to  which  a  software  product  is 
safe  from  unauthorized  use. 

Software  requirements  specification  -  A  document 
that  contains  the  complete  set  of  engineering  require¬ 
ments  for  each  computer  software  configuration  item. 

System  integration  -  The  activities  of  assembling 
hardware  and  software  components  into  a  deliverable 
product. 

Target  computer-  The  hardware  and  supporting  soft¬ 
ware  system  that  will  actually  be  used  when  the  soft¬ 
ware  system  is  fielded. 

TBDs  -  to  be  defined. 
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Test  specifications  -  A  document  that  prescribes  the 
process  and  procedures  to  be  used  to  verify  that  a 
product  meets  its  requirements. 

Traceability  mechanism-  Processes  and  procedures 
(manual  and/or  automated)  that  map  all  software  com¬ 
ponents  and  artifacts  from  source  requirements 
through  test  cases. 

Transition  plan  -  A  plan  (documented  in  the  Comput¬ 
er  Resources  Integrated  Support  document)  specifying 
how  products  are  to  transition  from  development  to 
support. 

Unit  testing- Testing  performed  by  the  developers  on 
an  individual  software  or  hardware  component  to  verify 
the  component  meets  its  allocated  requirements. 
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